apache/druid · error · FrameRowTooLargeException

FrameTooLarge

FrameTooLarge

Error message

FrameRowTooLargeException: row too large for frame allocator capacity %s

What it means

During window-function processing, writeRacToFrame attempts to append a row to the current frame writer and the frame's allocator has no room left even after the frame was (or could not be) flushed. A single row does not fit within the frame allocator capacity, so processing aborts with a FrameRowTooLargeException mapped to the FrameTooLarge fault code. This means an individual row (not the whole window) exceeds the memory budget allocated per frame.

Solutions

  1. Increase the MSQ frame allocator capacity (e.g. druid.msq.window.frame.size / frame-allocator memory limits in task properties)
  2. Reduce row width: select only needed columns, truncate/limit large string or array values before the window operator
  3. Increase worker/task memory (peon memory, task slot sizing) so larger frame capacities are allowed
  4. Split the query so window functions operate on fewer/narrower columns

Example fix

// before (task context)
{"maxRowsInMemory": 1000000, "druid.msq.window.frame.size": "256MiB"}
// after
{"druid.msq.window.frame.size": "1GiB"}
Defensive patterns

Strategy: validation

Validate before calling

// Estimate row size before window stage; ensure it fits frame capacity
long estimatedRowBytes = columns.stream().mapToLong(c -> c.estimatedMaxSize()).sum();
if (estimatedRowBytes >= frameAllocatorCapacityBytes) {
  throw new IllegalStateException("Row size " + estimatedRowBytes + " exceeds frame capacity " + frameAllocatorCapacityBytes);
}

Try / catch

try {
  runMsqQuery(query);
} catch (MSQException e) {
  if (e.getFault() instanceof FrameTooLarge) {
    // increase druid.msq.frame size or narrow projection and retry
  }
}

Prevention

When it happens

Trigger: A window-function stage writes a row whose serialized size exceeds frameWriterFactory.allocatorCapacity(); occurs when frameWriter.addSelection() returns false both after flushing and when the frame is empty (single row larger than capacity).

Common situations: Queries with very wide rows (many/large string columns), huge ARRAY/COMPLEX values, or maxRowsMaterialized/worker memory configured so the per-frame allocator capacity is smaller than one row; often after increasing row width via window function output columns.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/587d7887203f47f5. Report an issue: GitHub.

Appendix: source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/querykit/WindowOperatorQueryFrameProcessor.java:265

   * @throws IOException
   */
  public void writeRacToFrame(RowsAndColumns rac) throws IOException
  {
    final int numRows = rac.numRows();
    while (rowId.get() < numRows) {
      if (frameWriter.addSelection()) {
        incrementBoostColumn();
        rowId.incrementAndGet();
      } else if (frameWriter.getNumRows() > 0) {
        flushFrameWriter();
        createFrameWriterIfNeeded(rac);

        if (frameWriter.addSelection()) {
          incrementBoostColumn();
          rowId.incrementAndGet();
          return;
        } else {
          throw new FrameRowTooLargeException(frameWriterFactory.allocatorCapacity());
        }
      } else {
        throw new FrameRowTooLargeException(frameWriterFactory.allocatorCapacity());
      }
    }

    flushFrameWriter();
    clearRACBuffers();
  }

  @Override
  public void cleanup() throws IOException
  {
    FrameProcessors.closeAll(inputChannels(), outputChannels(), frameWriter);
  }

  /**
   * @return Number of rows flushed to the output channel

View on GitHub (pinned to 9b90983fd2)