apache/druid · error · FrameRowTooLargeException

FrameTooLarge

FrameTooLarge

Error message

FrameRowTooLargeException: row too large for frame allocator capacity %s

What it means

During window-function processing, writeRacToFrame attempts to append a row to the current frame writer and the frame's allocator has no room left even after the frame was (or could not be) flushed. A single row does not fit within the frame allocator capacity, so processing aborts with a FrameRowTooLargeException mapped to the FrameTooLarge fault code. This means an individual row (not the whole window) exceeds the memory budget allocated per frame.

Source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/querykit/WindowOperatorQueryFrameProcessor.java:265

   * @throws IOException
   */
  public void writeRacToFrame(RowsAndColumns rac) throws IOException
  {
    final int numRows = rac.numRows();
    while (rowId.get() < numRows) {
      if (frameWriter.addSelection()) {
        incrementBoostColumn();
        rowId.incrementAndGet();
      } else if (frameWriter.getNumRows() > 0) {
        flushFrameWriter();
        createFrameWriterIfNeeded(rac);

        if (frameWriter.addSelection()) {
          incrementBoostColumn();
          rowId.incrementAndGet();
          return;
        } else {
          throw new FrameRowTooLargeException(frameWriterFactory.allocatorCapacity());
        }
      } else {
        throw new FrameRowTooLargeException(frameWriterFactory.allocatorCapacity());
      }
    }

    flushFrameWriter();
    clearRACBuffers();
  }

  @Override
  public void cleanup() throws IOException
  {
    FrameProcessors.closeAll(inputChannels(), outputChannels(), frameWriter);
  }

  /**
   * @return Number of rows flushed to the output channel

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Increase the MSQ frame allocator capacity (e.g. druid.msq.window.frame.size / frame-allocator memory limits in task properties)
2. Reduce row width: select only needed columns, truncate/limit large string or array values before the window operator
3. Increase worker/task memory (peon memory, task slot sizing) so larger frame capacities are allowed
4. Split the query so window functions operate on fewer/narrower columns

Example fix

// before (task context)
{"maxRowsInMemory": 1000000, "druid.msq.window.frame.size": "256MiB"}
// after
{"druid.msq.window.frame.size": "1GiB"}

// Estimate row size before window stage; ensure it fits frame capacity
long estimatedRowBytes = columns.stream().mapToLong(c -> c.estimatedMaxSize()).sum();
if (estimatedRowBytes >= frameAllocatorCapacityBytes) {
  throw new IllegalStateException("Row size " + estimatedRowBytes + " exceeds frame capacity " + frameAllocatorCapacityBytes);
}

try {
  runMsqQuery(query);
} catch (MSQException e) {
  if (e.getFault() instanceof FrameTooLarge) {
    // increase druid.msq.frame size or narrow projection and retry
  }
}

When it happens

Trigger: A window-function stage writes a row whose serialized size exceeds frameWriterFactory.allocatorCapacity(); occurs when frameWriter.addSelection() returns false both after flushing and when the frame is empty (single row larger than capacity).

Common situations: Queries with very wide rows (many/large string columns), huge ARRAY/COMPLEX values, or maxRowsMaterialized/worker memory configured so the per-frame allocator capacity is smaller than one row; often after increasing row width via window function output columns.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

• TooManyRowsInAWindow
• FrameTooLarge
• NotEnoughMemoryFault
• FrameTooLarge
• Row too large to add to frame (max frame size = %,d)

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/587d7887203f47f5. Report an issue: GitHub.