apache/druid · critical · ColumnCapacityExceededException

Column capacity exceeded

Error message

Column capacity exceeded

What it means

CompressedColumnarIntsSerializer tracks the number of inserted values in an int; when numInserted overflows past Integer.MAX_VALUE it wraps negative and the serializer throws ColumnCapacityExceededException. Druid columns support at most Integer.MAX_VALUE rows, so this is a hard capacity limit, not a corruption.

Solutions

1. Reduce per-segment row count: set maxTotalRows / partitionsSpec so each segment stays well below Integer.MAX_VALUE rows.
2. Split the ingestion into multiple segments/tasks with finer partitioning.
3. If the column is a dictionary-encoded dimension, ensure the values written are the intended per-row values and not duplicated in a loop bug.

Example fix

// before: one segment for everything
// after: cap segment size
// tuning: maxTotalRows = 5_000_000 per shard so numInserted can never overflow

// Java
if (expectedRowCount >= Integer.MAX_VALUE) {
  throw new IllegalArgumentException("Segment too large; split into more partitions");
}

try {
  serializer.addValue(val);
} catch (ColumnCapacityExceededException e) {
  // abort and re-partition the shard
  throw new ISE("Row limit exceeded for segment; reduce maxTotalRows", e);
}

When it happens

Trigger: Calling addValue more than Integer.MAX_VALUE times on a single CompressedColumnarIntsSerializer (column with >2^31-1 values).

Common situations: Very large rollup-less datasets ingested into a single segment; maxStringDimensionLength/row count settings allowing segment sizes beyond the 2.1 billion row limit; ingestion jobs with partitioning configured too coarsely.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• Column capacity exceeded
• Column capacity exceeded
• Column capacity exceeded
• Column capacity exceeded
• Column capacity exceeded

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/29f6b04160a7b512. Report an issue: GitHub.

Appendix: source

Thrown at processing/src/main/java/org/apache/druid/segment/data/CompressedColumnarIntsSerializer.java:121

  {
    flattener.open();
  }

  @Override
  public void addValue(int val) throws IOException
  {
    if (endBuffer == null) {
      throw new IllegalStateException("written out already");
    }
    if (!endBuffer.hasRemaining()) {
      endBuffer.rewind();
      flattener.write(endBuffer);
      endBuffer.clear();
    }
    endBuffer.putInt(val);
    numInserted++;
    if (numInserted < 0) {
      throw new ColumnCapacityExceededException(columnName);
    }
  }

  @Override
  public long getSerializedSize() throws IOException
  {
    writeEndBuffer();
    return META_SERDE_HELPER.size(this) + flattener.getSerializedSize();
  }

  @Override
  public void writeTo(WritableByteChannel channel, SegmentFileBuilder fileBuilder) throws IOException
  {
    writeEndBuffer();
    META_SERDE_HELPER.writeTo(channel, this);
    flattener.writeTo(channel, fileBuilder);
  }

View on GitHub (pinned to 9b90983fd2)