{"record":{"id":"65ab22c2abbb1151","repo":"apache/druid","slug":"column-capacity-exceeded-65ab22","errorCode":null,"errorMessage":"Column capacity exceeded","messagePattern":"Column capacity exceeded","errorType":"exception","errorClass":"ColumnCapacityExceededException","httpStatus":null,"severity":"error","filePath":"processing/src/main/java/org/apache/druid/segment/data/EntireLayoutColumnarLongsSerializer.java","lineNumber":78,"sourceCode":"  public void open() throws IOException\n  {\n    valuesOut = segmentWriteOutMedium.makeWriteOutBytes();\n    writer.setOutputStream(valuesOut);\n  }\n\n  @Override\n  public int size()\n  {\n    return numInserted;\n  }\n\n  @Override\n  public void add(long value) throws IOException\n  {\n    writer.write(value);\n    ++numInserted;\n    if (numInserted < 0) {\n      throw new ColumnCapacityExceededException(columnName);\n    }\n  }\n\n  @Override\n  public long getSerializedSize() throws IOException\n  {\n    writer.flush();\n    return META_SERDE_HELPER.size(this) + valuesOut.size();\n  }\n\n  @Override\n  public void writeTo(WritableByteChannel channel, SegmentFileBuilder fileBuilder) throws IOException\n  {\n    writer.flush();\n    META_SERDE_HELPER.writeTo(channel, this);\n    valuesOut.writeTo(channel);\n  }\n}","sourceCodeStart":60,"sourceCodeEnd":96,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/processing/src/main/java/org/apache/druid/segment/data/EntireLayoutColumnarLongsSerializer.java#L60-L96","documentation":"Thrown by EntireLayoutColumnarLongsSerializer.add when numInserted overflows past Integer.MAX_VALUE. Long columns are int-indexed, so no serializer instance may hold more than 2^31 - 1 values; further writes would overflow the internal int counter. ColumnCapacityExceededException fails the ingestion rather than writing a corrupt column.","triggerScenarios":"Calling add(long) on a serializer that has already accepted Integer.MAX_VALUE values; numInserted becomes negative and the check fires.","commonSituations":"Ingestion jobs that produce more than 2 billion rows into a single segment due to misconfigured partitioning; writing large synthetic datasets in one shot; row-limit tuning configs set to maximum.","solutions":["Partition the input so each segment stays below Integer.MAX_VALUE rows (tune maxRowsPerSegment, maxTotalRows)","Lower maxRowsInMemory to force frequent segment persists","Use hash/dynamic partitioning with a suitable partition dimension","Split oversized inputs across multiple tasks"],"exampleFix":"// before\nmaxRowsPerSegment = 5000000000L; // exceeds int-indexed column capacity\n// after\nmaxRowsPerSegment = 5000000; // each segment column stays far below 2^31","handlingStrategy":"validation","validationCode":"if (rowsInSegment >= Integer.MAX_VALUE) {\n  throw new IllegalStateException(\"Long column exceeds max int-indexed capacity\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  serializer.add(value);\n} catch (ColumnCapacityExceededException e) {\n  throw new RuntimeException(\"Long column capacity exceeded; repartition data\", e);\n}","preventionTips":["Configure maxRowsPerSegment/maxTotalRows to cap segment size","Split huge datasets across multiple ingestion tasks","Alert on per-segment row counts during backfills"],"tags":["capacity","segment","serialization"],"backgroundTag":"value-out-of-range","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}