{"record":{"id":"29f6b04160a7b512","repo":"apache/druid","slug":"column-capacity-exceeded-29f6b0","errorCode":null,"errorMessage":"Column capacity exceeded","messagePattern":"Column capacity exceeded","errorType":"exception","errorClass":"ColumnCapacityExceededException","httpStatus":null,"severity":"critical","filePath":"processing/src/main/java/org/apache/druid/segment/data/CompressedColumnarIntsSerializer.java","lineNumber":121,"sourceCode":"  {\n    flattener.open();\n  }\n\n  @Override\n  public void addValue(int val) throws IOException\n  {\n    if (endBuffer == null) {\n      throw new IllegalStateException(\"written out already\");\n    }\n    if (!endBuffer.hasRemaining()) {\n      endBuffer.rewind();\n      flattener.write(endBuffer);\n      endBuffer.clear();\n    }\n    endBuffer.putInt(val);\n    numInserted++;\n    if (numInserted < 0) {\n      throw new ColumnCapacityExceededException(columnName);\n    }\n  }\n\n  @Override\n  public long getSerializedSize() throws IOException\n  {\n    writeEndBuffer();\n    return META_SERDE_HELPER.size(this) + flattener.getSerializedSize();\n  }\n\n  @Override\n  public void writeTo(WritableByteChannel channel, SegmentFileBuilder fileBuilder) throws IOException\n  {\n    writeEndBuffer();\n    META_SERDE_HELPER.writeTo(channel, this);\n    flattener.writeTo(channel, fileBuilder);\n  }\n","sourceCodeStart":103,"sourceCodeEnd":139,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/processing/src/main/java/org/apache/druid/segment/data/CompressedColumnarIntsSerializer.java#L103-L139","documentation":"CompressedColumnarIntsSerializer tracks the number of inserted values in an int; when numInserted overflows past Integer.MAX_VALUE it wraps negative and the serializer throws ColumnCapacityExceededException. Druid columns support at most Integer.MAX_VALUE rows, so this is a hard capacity limit, not a corruption.","triggerScenarios":"Calling addValue more than Integer.MAX_VALUE times on a single CompressedColumnarIntsSerializer (column with >2^31-1 values).","commonSituations":"Very large rollup-less datasets ingested into a single segment; maxStringDimensionLength/row count settings allowing segment sizes beyond the 2.1 billion row limit; ingestion jobs with partitioning configured too coarsely.","solutions":["Reduce per-segment row count: set maxTotalRows / partitionsSpec so each segment stays well below Integer.MAX_VALUE rows.","Split the ingestion into multiple segments/tasks with finer partitioning.","If the column is a dictionary-encoded dimension, ensure the values written are the intended per-row values and not duplicated in a loop bug."],"exampleFix":"// before: one segment for everything\n// after: cap segment size\n// tuning: maxTotalRows = 5_000_000 per shard so numInserted can never overflow\n","handlingStrategy":"validation","validationCode":"// Java\nif (expectedRowCount >= Integer.MAX_VALUE) {\n  throw new IllegalArgumentException(\"Segment too large; split into more partitions\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  serializer.addValue(val);\n} catch (ColumnCapacityExceededException e) {\n  // abort and re-partition the shard\n  throw new ISE(\"Row limit exceeded for segment; reduce maxTotalRows\", e);\n}","preventionTips":["Set maxTotalRows/partitionsSpec well below 2^31 rows per segment.","Monitor per-segment row counts in ingestion metrics.","Account for multi-value rows inflating total inserted values."],"tags":["capacity","overflow","segment-limits"],"backgroundTag":"value-out-of-range","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}