{"record":{"id":"955cfa2c676e4d8f","repo":"apache/druid","slug":"column-capacity-exceeded","errorCode":null,"errorMessage":"Column capacity exceeded","messagePattern":"Column capacity exceeded","errorType":"exception","errorClass":"ColumnCapacityExceededException","httpStatus":null,"severity":"error","filePath":"processing/src/main/java/org/apache/druid/segment/data/BlockLayoutColumnarDoublesSerializer.java","lineNumber":104,"sourceCode":"    return numInserted;\n  }\n\n  @Override\n  public void add(double value) throws IOException\n  {\n    if (endBuffer == null) {\n      throw new IllegalStateException(\"written out already\");\n    }\n    if (!endBuffer.hasRemaining()) {\n      endBuffer.rewind();\n      flattener.write(endBuffer);\n      endBuffer.clear();\n    }\n\n    endBuffer.putDouble(value);\n    ++numInserted;\n    if (numInserted < 0) {\n      throw new ColumnCapacityExceededException(columnName);\n    }\n  }\n\n  @Override\n  public long getSerializedSize() throws IOException\n  {\n    writeEndBuffer();\n    return META_SERDE_HELPER.size(this) + flattener.getSerializedSize();\n  }\n\n  @Override\n  public void writeTo(WritableByteChannel channel, SegmentFileBuilder fileBuilder) throws IOException\n  {\n    writeEndBuffer();\n    META_SERDE_HELPER.writeTo(channel, this);\n    flattener.writeTo(channel, fileBuilder);\n  }\n","sourceCodeStart":86,"sourceCodeEnd":122,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/processing/src/main/java/org/apache/druid/segment/data/BlockLayoutColumnarDoublesSerializer.java#L86-L122","documentation":"ColumnCapacityExceededException is thrown when a Druid segment column serializer has inserted more rows than a column can address. The serializers track row count in an int; once numInserted overflows to negative, the column's internal offset encoding can no longer represent the row count, so writing further would corrupt the segment. Druid throws this instead of silently producing a corrupt segment.","triggerScenarios":"Calling add(double) on BlockLayoutColumnarDoublesSerializer after 2^31 rows (Integer.MAX_VALUE) have been inserted into a single column, so ++numInserted wraps to a negative value.","commonSituations":"Very large batch ingestion jobs or long-running streaming ingestion that pushes more than ~2.1 billion rows into a single segment without segment granularity/maxRowsPerSegment tuning; version changes that change row-count encoding.","solutions":["Split the data into more segments (lower maxRowsPerSegment / use finer segmentGranularity) so no column exceeds Integer.MAX_VALUE rows","Check ingestion specs for overly large maxRowsPerSegment or maxTotalRows settings and reduce them","Ensure roll-up/deduplication or partitioning (e.g., hash partitioning) is used to cap per-segment row counts","If you truly need >2B rows per column, upgrade to a Druid version/segment format that raises the limit"],"exampleFix":"// before: one giant segment\n// ingestion spec without row caps\n// after\n\"partitionsSpec\": {\n  \"type\": \"hashed\",\n  \"maxRowsPerSegment\": 5000000\n}","handlingStrategy":"validation","validationCode":"// before writing, cap rows per segment\nif (rowCount >= Integer.MAX_VALUE - 1) {\n  throw new IllegalArgumentException(\"Segment row count would exceed column capacity; cut segment first\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  serializer.add(value);\n} catch (ColumnCapacityExceededException e) {\n  // cut current segment, start a new one, retry the row there\n  rollOverToNewSegment(e.getColumnName());\n}","preventionTips":["Set maxRowsPerSegment well below 2^31","Use time or hash partitioning to distribute rows","Enable roll-up to reduce row counts","Monitor per-segment row counts during ingestion"],"tags":["druid","segment","ingestion","capacity","overflow"],"backgroundTag":"value-out-of-range","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}