apache/druid · error · FrameRowTooLargeException

Row too large to add to frame (max frame size = %,d)

Error message

Row too large to add to frame (max frame size = %,d)

What it means

ScanQueryFrameProcessor.populateFrameWriterAndFlushIfNeeded throws FrameRowTooLargeException when frameWriter.addSelection() fails and the current frame is still empty, meaning a single scan result row exceeds the frame's allocator capacity. MSQ cannot split a row across frames, so the processor aborts.

Source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/querykit/scan/ScanQueryFrameProcessor.java:491

  private void populateFrameWriterAndFlushIfNeeded() throws IOException
  {
    createFrameWriterIfNeeded();

    while (!cursor.isDone()) {
      boolean flush;

      if (frameWriter.addSelection()) {
        cursorRowsRead++;
        cursor.advance();
        cursorOffset.increment();
        partitionBoostVirtualColumn.setValue(partitionBoostVirtualColumn.getValue() + 1);

        // Flush if we reached cursorPushDownLimit.
        flush = cursorPushDownLimit >= 0 && cursorRowsRead >= cursorPushDownLimit;
      } else {
        // addSelection failed because the frame is full.
        if (frameWriter.getNumRows() == 0) {
          throw new FrameRowTooLargeException(currentAllocatorCapacity);
        }

        flush = true;
      }

      if (flush) {
        final long numRowsWritten = flushFrameWriter();

        if (sharedRunningCountForLimit != null) {
          sharedRunningCountForLimit.addAndGet(numRowsWritten);
        }

        break;
      }
    }
  }

  private void createFrameWriterIfNeeded()

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Increase MSQ frame allocator capacity / worker memory configuration.
2. Select only needed columns to shrink each row.
3. Fix upstream ingestion to avoid storing oversized rows.
4. Reduce per-task concurrency to allow larger frames within the same memory budget.

Example fix

// before
SELECT * FROM huge_table  // row > frame capacity
// after
SELECT col1, col2 FROM huge_table  // narrower rows fit the frame

// Pre-check: estimated scan row size vs frame capacity
if (selectedColumnsBytesEstimate > frameCapacityBytes) {
  throw new Error('Scan row exceeds frame capacity; select fewer columns or raise frame memory');
}

try {
  runMsqScan(scan);
} catch (FrameRowTooLargeException e) {
  // Retry with a narrower column list or larger frame allocation
  retry(selectSubset(scan, e.getMaxFrameSize()));
}

When it happens

Trigger: Scanning segments or input frames via MSQ where one row (all selected columns serialized) is larger than the current frame allocator capacity — wide selects, long string values, or undersized frame memory config.

Common situations: SELECT * on wide datasources with large strings; ingestion of oversized rows later scanned with MSQ; MSQ workers with low frame memory limits.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

• Row too large to add to frame (max frame size = %,d)
• Row too large to add to frame (max frame size = %,d)
• Null cursor factory found. Probably trying to issue a query
• NotEnoughMemoryFault
• FrameTooLarge

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/a88c46883480df8c. Report an issue: GitHub.