apache/druid · critical · ColumnCapacityExceededException

Column capacity exceeded

Error message

Column capacity exceeded

What it means

CompressedVSizeColumnarIntsSerializer increments numInserted per addValue call and throws ColumnCapacityExceededException when the int counter overflows (numInserted < 0), i.e. more than Integer.MAX_VALUE values were inserted. Druid columns cannot exceed ~2.1 billion entries.

Solutions

1. Configure partitionsSpec/maxTotalRows so segments stay far below 2^31 rows.
2. For multi-value dimensions, reduce the number of values per row or shard the data.
3. Split the workload across more ingestion tasks/segments.

Example fix

// before: dynamic partitioning with no maxTotalRows
// after
"partitionsSpec": { "type": "hashed", "maxTotalRows": 5000000 }

// Java
if ((long) numExpectedValues >= Integer.MAX_VALUE) {
  throw new IllegalArgumentException("Column would exceed 2^31 values; re-partition");
}

try {
  serializer.addValue(val);
} catch (ColumnCapacityExceededException e) {
  throw new ISE("Column capacity exceeded; split segment", e);
}

When it happens

Trigger: Adding more than Integer.MAX_VALUE values to a single CompressedVSizeColumnarIntsSerializer - typically a column with more than 2^31-1 rows (or values for multi-value rows) in one segment.

Common situations: Oversized segments from coarse partitioning; huge multi-value dimensions whose total value count exceeds the limit even when row count does not; ingestion tasks without maxTotalRows bounds.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• Column capacity exceeded
• Column capacity exceeded
• Column capacity exceeded
• Column capacity exceeded
• Column capacity exceeded

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/ccd330adf28467d1. Report an issue: GitHub.

Appendix: source

Thrown at processing/src/main/java/org/apache/druid/segment/data/CompressedVSizeColumnarIntsSerializer.java:167

  public void addValue(int val) throws IOException
  {
    if (endBuffer == null) {
      throw new IllegalStateException("written out already");
    }
    if (!endBuffer.hasRemaining()) {
      endBuffer.rewind();
      flattener.write(endBuffer);
      endBuffer.clear();
    }
    intBuffer.putInt(0, val);
    if (isBigEndian) {
      endBuffer.put(intBuffer.array(), Integer.BYTES - numBytes, numBytes);
    } else {
      endBuffer.put(intBuffer.array(), 0, numBytes);
    }
    numInserted++;
    if (numInserted < 0) {
      throw new ColumnCapacityExceededException(columnName);
    }
  }

  @Override
  public long getSerializedSize() throws IOException
  {
    writeEndBuffer();
    return META_SERDE_HELPER.size(this) + flattener.getSerializedSize();
  }

  @Override
  public void writeTo(WritableByteChannel channel, SegmentFileBuilder fileBuilder) throws IOException
  {
    writeEndBuffer();
    META_SERDE_HELPER.writeTo(channel, this);
    flattener.writeTo(channel, fileBuilder);
  }

View on GitHub (pinned to 9b90983fd2)