apache/druid · error · MSQException

TooManyRowsInAWindow

TooManyRowsInAWindow

Error message

TooManyRowsInAWindowFault: too many rows in a window, %s > max %s

What it means

MSQ window processing materializes the rows of a window partition in memory and enforces a cap (maxRowsMaterialized). ensureMaxRowsInAWindowConstraint throws TooManyRowsInAWindowFault when a single window partition contains more rows than this configured maximum. The query is deliberately stopped instead of exhausting worker memory.

Source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/querykit/WindowOperatorQueryFrameProcessor.java:328

    final RowSignature signature = frameReader.signature();
    RowBasedFrameRowsAndColumns frameRowsAndColumns = new RowBasedFrameRowsAndColumns(frame, signature);
    LazilyDecoratedRowsAndColumns ldrc = new LazilyDecoratedRowsAndColumns(
        frameRowsAndColumns,
        null,
        null,
        null,
        OffsetLimit.limit(Integer.MAX_VALUE),
        null,
        null,
        frameWriterFactory.allocatorCapacity()
    );
    return ldrc;
  }

  private static void ensureMaxRowsInAWindowConstraint(int numRowsInWindow, int maxRowsMaterialized)
  {
    if (numRowsInWindow > maxRowsMaterialized) {
      throw new MSQException(new TooManyRowsInAWindowFault(
          numRowsInWindow,
          maxRowsMaterialized
      ));
    }
  }

  private boolean needToProcessBatch()
  {
    return frameRowsAndColsBuilder.getNumRows() >= maxRowsMaterialized / 2; // Can this be improved further?
  }

  private static class RowsAndColumnsBuilder
  {
    private final List<RowsAndColumns> racList;
    private int totalRows;
    private final int maxRowsMaterialized;

    public RowsAndColumnsBuilder(int maxRowsMaterialized)

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Increase the context parameter maxRowsInWindow / maxRowsMaterialized for the query (mind memory)
2. Add more columns to PARTITION BY so each window partition contains fewer rows
3. Pre-aggregate rows before applying the window function
4. Split the data (e.g. by time or additional key) into smaller window partitions

Example fix

// before
"context": {"maxRowsInWindow": 100000}
// after
"context": {"maxRowsInWindow": 1000000}

// Check partition cardinality against maxRowsInWindow before submitting
long maxPartitionRows = estimateMaxRowsPerPartition(table, partitionKeys);
if (maxPartitionRows > maxRowsInWindow) {
  throw new IllegalStateException("Partition has " + maxPartitionRows + " rows > maxRowsInWindow " + maxRowsInWindow);
}

try {
  runMsqQuery(query);
} catch (MSQException e) {
  if (e.getFault() instanceof TooManyRowsInAWindow) {
    TooManyRowsInAWindowFault f = (TooManyRowsInAWindowFault) e.getFault();
    // resubmit with maxRowsInWindow >= f.getNumRowsInWindow() or refine PARTITION BY
  }
}

When it happens

Trigger: A window function (OVER clause) whose partition-by key groups so many rows that numRowsInWindow exceeds maxRowsInWindow/maxRowsMaterialized (default typically 100,000), checked from goOrContinue.

Common situations: A partition column with very low cardinality (e.g. all rows share one partition key) while running window functions over a large table; copying example context settings that cap rows in a window.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• FrameTooLarge
• NotEnoughMemoryFault
• TooManyInputFiles
• FrameTooLarge
• FrameTooLarge

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/3650fdef8ebfaa33. Report an issue: GitHub.