apache/druid · error · FrameRowTooLargeException

Row too large to add to frame (max frame size = %,d)

Error message

Row too large to add to frame (max frame size = %,d)

What it means

GroupByPreShuffleFrameProcessor.populateFrameWriterAndFlushIfNeeded throws FrameRowTooLargeException when a single pre-shuffle group-by result row cannot fit into a completely empty frame (frameWriter.getNumRows() == 0 and addSelection() fails). MSQ frames cannot hold a row larger than the frame's allocator capacity, so processing aborts.

Source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/querykit/groupby/GroupByPreShuffleFrameProcessor.java:328

  public void cleanup() throws IOException
  {
    closer.register(this::closeAndDiscardResultYielder);
    closer.register(frameWriter);
    closer.register(super::cleanup);
    closer.close();
  }

  private void populateFrameWriterAndFlushIfNeeded() throws IOException
  {
    createFrameWriterIfNeeded();

    while (!resultYielder.isDone()) {
      final boolean didAddToFrame = frameWriter.addSelection();

      if (didAddToFrame) {
        resultYielder = resultYielder.next(null);
      } else if (frameWriter.getNumRows() == 0) {
        throw new FrameRowTooLargeException(currentAllocatorCapacity);
      } else {
        flushFrameWriterIfNeeded();
        return;
      }
    }

    flushFrameWriterIfNeeded();
    closeAndDiscardResultYielder();
  }

  private void createFrameWriterIfNeeded()
  {
    if (frameWriter == null) {
      final FrameWriterFactory frameWriterFactory = getFrameWriterFactory();
      frameWriter = frameWriterFactory.newFrameWriter(frameWriterColumnSelectorFactory);
      currentAllocatorCapacity = frameWriterFactory.allocatorCapacity();
    }
  }

View on GitHub (pinned to 9b90983fd2)

Solutions

  1. Increase frame memory/allocation in MSQ worker configuration so the largest row fits.
  2. Reduce group-by key width: fewer dimensions, shorter strings, hashed keys.
  3. Use more compact aggregators or reduce aggregator count.
  4. Increase per-task worker memory or lower concurrency to free memory for frames.

Example fix

// before
// worker: 1GiB memory, default frame size -> row of 2MB does not fit
// after
// raise frame capacity or worker memory, e.g. give each MSQ worker 4GiB and raise druid.msq.frame allocation
Defensive patterns

Strategy: try-catch

Try / catch

try {
  runMsqQuery(groupBy);
} catch (FrameRowTooLargeException e) {
  // Retry with reduced key width or increased frame capacity
  retry(withNarrowerKeys(withLargerFrames(groupBy, e.getMaxFrameSize())));
}

Prevention

When it happens

Trigger: Pre-shuffle group-by execution where one output row's serialized size exceeds the current frame allocator capacity; triggered when reading from a data server query, segment, or input channel yields rows too wide for the configured frame size.

Common situations: Very wide dimension sets or long strings in group-by keys; sketch aggregators with large state; MSQ workers configured with too little frame memory (druid.msq.frame settings).

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/2f8c587e724c6be5. Report an issue: GitHub.