{"record":{"id":"c75ca7e384ecf4d8","repo":"apache/druid","slug":"columncapacityexceededexception","errorCode":null,"errorMessage":"ColumnCapacityExceededException","messagePattern":"ColumnCapacityExceededException","errorType":"exception","errorClass":"ColumnCapacityExceededException","httpStatus":null,"severity":"critical","filePath":"extensions-contrib/spectator-histogram/src/main/java/org/apache/druid/spectator/histogram/SpectatorHistogramSerializer.java","lineNumber":87,"sourceCode":"  {\n    this.columnName = columnName;\n    this.segmentWriteOutMedium = segmentWriteOutMedium;\n    this.objectStrategy = strategy;\n  }\n\n  @Override\n  public void open() throws IOException\n  {\n    this.offsetsHeader = NullableOffsetsHeader.create(segmentWriteOutMedium);\n    this.valuesBuffer = segmentWriteOutMedium.makeWriteOutBytes();\n  }\n\n  @Override\n  public void serialize(ColumnValueSelector<?> selector) throws IOException\n  {\n    rowCount++;\n    if (rowCount < 0) {\n      throw new ColumnCapacityExceededException(columnName);\n    }\n    Object value = selector.getObject();\n    if (value == null) {\n      offsetsHeader.writeNull();\n    } else {\n      objectStrategy.writeTo((SpectatorHistogram) value, valuesBuffer);\n      offsetsHeader.writeOffset(Ints.checkedCast(valuesBuffer.size()));\n    }\n  }\n\n  @Override\n  public long getSerializedSize()\n  {\n    // Meta header, Offsets, Values\n    return META_SERDE_HELPER.size(this) + offsetsHeader.getSerializedSize() + valuesBuffer.size();\n  }\n\n  @Override","sourceCodeStart":69,"sourceCodeEnd":105,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-contrib/spectator-histogram/src/main/java/org/apache/druid/spectator/histogram/SpectatorHistogramSerializer.java#L69-L105","documentation":"During column serialization, each call to serialize() increments rowCount; if rowCount overflows past Integer.MAX_VALUE (rowCount becomes negative), the writer throws ColumnCapacityExceededException because the column cannot hold more rows than an int can count.","triggerScenarios":"Ingesting more than ~2.1 billion rows into a single spectator-histogram column segment; rowCount wraps to negative on the 2^31st row and serialize() throws.","commonSituations":"Very large batch tasks without partitioning; misconfigured maxRowNumInSegment allowing unbounded segment growth; ingestion of an entire huge table into one segment.","solutions":["Reduce rows per segment: lower druid.segment.rowNumTarget / maxNumSegmentPartitions or tune partitionsSpec so segments stay well below Integer.MAX_VALUE rows.","Re-partition the ingestion task input so no single task writes >2^31 rows.","If thrown, split the input data and run separate ingestion tasks.","Verify no counter bug is causing repeated serialize calls for the same rows."],"exampleFix":"// before (tuning)\n// single task ingesting unbounded rows\n// after\ndynamic partitionsSpec with maxTotalRows well below 2_000_000_000, or ranged partitions on time","handlingStrategy":"validation","validationCode":"if (taskMaxRows > Integer.MAX_VALUE - 1000) { throw new IllegalArgumentException(\"segment row budget too close to Integer.MAX_VALUE; reduce partitions\"); }","typeGuard":null,"tryCatchPattern":"try { serializer.serialize(selector); } catch (ColumnCapacityExceededException e) { throw new IOException(\"segment exceeded row capacity; re-partition input: \" + e.getMessage(), e); }","preventionTips":["Keep segments to millions of rows, far below 2^31.","Tune partitionsSpec / maxTotalRows for large ingests.","Split oversized input datasets across multiple tasks.","Monitor rowCount in ingestion logs for runaway growth."],"tags":["java","druid","ingestion","segment-size"],"backgroundTag":"file-size-limit-exceeded","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}