{"record":{"id":"752b3543843b6b9c","repo":"apache/druid","slug":"column-capacity-exceeded-752b35","errorCode":null,"errorMessage":"Column capacity exceeded","messagePattern":"Column capacity exceeded","errorType":"exception","errorClass":"ColumnCapacityExceededException","httpStatus":null,"severity":"error","filePath":"processing/src/main/java/org/apache/druid/segment/data/CompressedBlockSerializer.java","lineNumber":164,"sourceCode":"  }\n\n  private void flushBuffer() throws IOException\n  {\n    uncompressedDataBuffer.rewind();\n    compressedDataBuffer.clear();\n\n    final ByteBuffer compressed = compressor.compress(uncompressedDataBuffer, compressedDataBuffer);\n\n    currentOffset += compressed.remaining();\n    offsetValueConverter.clear();\n    offsetValueConverter.putInt(currentOffset);\n    offsetValueConverter.flip();\n    Channels.writeFully(headerOut, offsetValueConverter);\n    Channels.writeFully(valuesOut, compressed);\n    uncompressedDataBuffer.clear();\n    numBlocks++;\n    if (numBlocks < 0) {\n      throw new ColumnCapacityExceededException(\"compressed\");\n    }\n  }\n\n  private void writeEndBuffer() throws IOException\n  {\n    if (uncompressedDataBuffer != null) {\n      uncompressedDataBuffer.flip();\n      flushBuffer();\n      uncompressedDataBuffer = null;\n    }\n  }\n}\n","sourceCodeStart":146,"sourceCodeEnd":177,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/processing/src/main/java/org/apache/druid/segment/data/CompressedBlockSerializer.java#L146-L177","documentation":"CompressedBlockSerializer.flushBuffer throws ColumnCapacityExceededException when the block count (numBlocks, an int) overflows to negative after incrementing. The compressed block format addresses blocks with an int, so exceeding Integer.MAX_VALUE blocks means the column cannot encode more compressed blocks and Druid aborts instead of corrupting the segment.","triggerScenarios":"Flushing enough compressed blocks from a very large column that numBlocks exceeds 2^31-1 — extremely large columns with tiny per-block payloads or billions of values.","commonSituations":"Giant single-column segments; misconfigured block sizes causing very frequent flushes; unpartitioned mass ingestion.","solutions":["Split the column/segment before the block count nears Integer.MAX_VALUE (reduce maxRowsPerSegment)","Increase compression block size so fewer blocks are produced for the same data","Partition the data (time/hash) to cap per-segment column size","Verify with ingestion metrics which column hit the limit and adjust the spec"],"exampleFix":"// before: tiny blocks -> too many blocks\n\"compression\": { \"blockSize\": 64 }\n// after\n\"compression\": { \"blockSize\": 8192 } // plus maxRowsPerSegment cap","handlingStrategy":"validation","validationCode":"// estimate blocks: values / blockSize; refuse before overflow\nif ((long) estimatedValues / blockSize >= Integer.MAX_VALUE) {\n  throw new IllegalArgumentException(\"Column would exceed compressed block capacity; split segment\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  flushBuffer();\n} catch (ColumnCapacityExceededException e) {\n  logger.error(\"Compressed block count overflow; abort segment and split input\");\n  throw new SegmentBuildException(\"Split input into more segments\", e);\n}","preventionTips":["Increase compression block size to reduce block count","Cap segment rows with maxRowsPerSegment","Partition inputs before ingestion","Alert on growing column sizes in ingestion metrics"],"tags":["druid","segment","capacity","overflow","compression"],"backgroundTag":"value-out-of-range","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}