apache/druid · error · ColumnCapacityExceededException

Column capacity exceeded

Error message

Column capacity exceeded

What it means

CompressedBlockSerializer.flushBuffer throws ColumnCapacityExceededException when the block count (numBlocks, an int) overflows to negative after incrementing. The compressed block format addresses blocks with an int, so exceeding Integer.MAX_VALUE blocks means the column cannot encode more compressed blocks and Druid aborts instead of corrupting the segment.

Source

Thrown at processing/src/main/java/org/apache/druid/segment/data/CompressedBlockSerializer.java:164

  }

  private void flushBuffer() throws IOException
  {
    uncompressedDataBuffer.rewind();
    compressedDataBuffer.clear();

    final ByteBuffer compressed = compressor.compress(uncompressedDataBuffer, compressedDataBuffer);

    currentOffset += compressed.remaining();
    offsetValueConverter.clear();
    offsetValueConverter.putInt(currentOffset);
    offsetValueConverter.flip();
    Channels.writeFully(headerOut, offsetValueConverter);
    Channels.writeFully(valuesOut, compressed);
    uncompressedDataBuffer.clear();
    numBlocks++;
    if (numBlocks < 0) {
      throw new ColumnCapacityExceededException("compressed");
    }
  }

  private void writeEndBuffer() throws IOException
  {
    if (uncompressedDataBuffer != null) {
      uncompressedDataBuffer.flip();
      flushBuffer();
      uncompressedDataBuffer = null;
    }
  }
}

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Split the column/segment before the block count nears Integer.MAX_VALUE (reduce maxRowsPerSegment)
2. Increase compression block size so fewer blocks are produced for the same data
3. Partition the data (time/hash) to cap per-segment column size
4. Verify with ingestion metrics which column hit the limit and adjust the spec

Example fix

// before: tiny blocks -> too many blocks
"compression": { "blockSize": 64 }
// after
"compression": { "blockSize": 8192 } // plus maxRowsPerSegment cap

// estimate blocks: values / blockSize; refuse before overflow
if ((long) estimatedValues / blockSize >= Integer.MAX_VALUE) {
  throw new IllegalArgumentException("Column would exceed compressed block capacity; split segment");
}

try {
  flushBuffer();
} catch (ColumnCapacityExceededException e) {
  logger.error("Compressed block count overflow; abort segment and split input");
  throw new SegmentBuildException("Split input into more segments", e);
}

When it happens

Trigger: Flushing enough compressed blocks from a very large column that numBlocks exceeds 2^31-1 — extremely large columns with tiny per-block payloads or billions of values.

Common situations: Giant single-column segments; misconfigured block sizes causing very frequent flushes; unpartitioned mass ingestion.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• Column capacity exceeded
• Column capacity exceeded
• Column capacity exceeded
• Cannot apply limit[%d] with offset[%d] due to overflow
• File[%s] too large[%d]

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/752b3543843b6b9c. Report an issue: GitHub.