{"record":{"id":"c9de4eb1df59e552","repo":"apache/druid","slug":"row-too-large-to-add-to-frame-max-frame-size","errorCode":null,"errorMessage":"Row too large to add to frame (max frame size = %,d)","messagePattern":"Row too large to add to frame \\(max frame size = %,d\\)","errorType":"exception","errorClass":"FrameRowTooLargeException","httpStatus":null,"severity":"error","filePath":"multi-stage-query/src/main/java/org/apache/druid/msq/querykit/groupby/GroupByPostShuffleFrameProcessor.java","lineNumber":255,"sourceCode":"\n    // Finalize aggregators after checking if they are passing the havingSpec, because havingSpec expects the\n    // unfinalized row (and finalizes it internally after making a copy of it)\n    finalizeFn.accept(outputRow);\n\n    if (frameWriter.addSelection()) {\n      incrementBoostColumn();\n      outputRow = null;\n      return false;\n    } else if (frameWriter.getNumRows() > 0) {\n      writeCurrentFrameIfNeeded();\n      setUpFrameWriterIfNeeded();\n\n      if (frameWriter.addSelection()) {\n        incrementBoostColumn();\n        outputRow = null;\n        return true;\n      } else {\n        throw new FrameRowTooLargeException(frameWriterFactory.allocatorCapacity());\n      }\n    } else {\n      throw new FrameRowTooLargeException(frameWriterFactory.allocatorCapacity());\n    }\n  }\n\n  private void writeCurrentFrameIfNeeded() throws IOException\n  {\n    if (frameWriter != null && frameWriter.getNumRows() > 0) {\n      final Frame frame = Frame.wrap(frameWriter.toByteArray());\n      outputChannel.write(frame);\n      frameWriter.close();\n      frameWriter = null;\n      outputRows += frame.numRows();\n    }\n  }\n\n  /**","sourceCodeStart":237,"sourceCodeEnd":273,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/multi-stage-query/src/main/java/org/apache/druid/msq/querykit/groupby/GroupByPostShuffleFrameProcessor.java#L237-L273","documentation":"GroupByPostShuffleFrameProcessor.writeOutputRow throws FrameRowTooLargeException when a single aggregation output row cannot be added to the current frame because it exceeds the frame writer's allocator capacity. MSQ frames have a fixed maximum size, and one row that does not fit in an empty frame cannot be split across frames.","triggerScenarios":"A group-by query executed via MSQ produces a post-shuffle output row whose serialized size (dimensions + aggregators, including any boost/virtual columns) exceeds the configured max frame size (druid.msq.frame.maxRows or memory allocation per frame).","commonSituations":"Queries with very wide result rows (many dimensions, large string values, complex sketches like HLL/quantiles), or overly small frame memory allocation in the MSQ worker config.","solutions":["Increase per-frame capacity: raise the frame memory allocation / maxRows config (druid.msq.* frame settings) so a row fits.","Reduce row width: select fewer dimensions/columns, shorten string cardinality, or use smaller sketch types.","Increase worker memory or reduce worker capacity so more memory is available for frames.","Split the query (e.g. aggregate in stages with fewer output columns) to shrink each row."],"exampleFix":"// before (druid config)\ndruid.msq.frame.maxRows=100000  // frames too small for wide rows\n// after\ndruid.msq.frame.maxRows=10000  // smaller rows-per-frame => larger per-row capacity with same memory","handlingStrategy":"try-catch","validationCode":"// Estimate row size against frame capacity before running wide group-bys\nconst estimatedRowBytes = dimCount * 64 + aggCount * 256;\nif (estimatedRowBytes > frameCapacityBytes) throw new Error('Row likely exceeds MSQ frame capacity; increase frame memory or reduce row width');","typeGuard":null,"tryCatchPattern":"try {\n  runMsqQuery(query);\n} catch (FrameRowTooLargeException e) {\n  // e reports max frame size; retry with fewer columns or bigger frames\n  retryWithWiderFramesOrNarrowerQuery(query, e.getMaxFrameSize());\n}","preventionTips":["Keep group-by output rows narrow (few/small dimensions and aggregators).","Size frame memory to comfortably exceed your largest expected row.","Use compact sketch aggregators with appropriate lgK.","Load-test wide queries against the configured frame size."],"tags":["msq","memory","frame"],"backgroundTag":"payload-too-large","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}