apache/druid · error · ColumnCapacityExceededException

Column capacity exceeded

Error message

Column capacity exceeded

What it means

ColumnCapacityExceededException is thrown when a Druid segment column serializer has inserted more rows than a column can address. The serializers track row count in an int; once numInserted overflows to negative, the column's internal offset encoding can no longer represent the row count, so writing further would corrupt the segment. Druid throws this instead of silently producing a corrupt segment.

Source

Thrown at processing/src/main/java/org/apache/druid/segment/data/BlockLayoutColumnarDoublesSerializer.java:104

    return numInserted;
  }

  @Override
  public void add(double value) throws IOException
  {
    if (endBuffer == null) {
      throw new IllegalStateException("written out already");
    }
    if (!endBuffer.hasRemaining()) {
      endBuffer.rewind();
      flattener.write(endBuffer);
      endBuffer.clear();
    }

    endBuffer.putDouble(value);
    ++numInserted;
    if (numInserted < 0) {
      throw new ColumnCapacityExceededException(columnName);
    }
  }

  @Override
  public long getSerializedSize() throws IOException
  {
    writeEndBuffer();
    return META_SERDE_HELPER.size(this) + flattener.getSerializedSize();
  }

  @Override
  public void writeTo(WritableByteChannel channel, SegmentFileBuilder fileBuilder) throws IOException
  {
    writeEndBuffer();
    META_SERDE_HELPER.writeTo(channel, this);
    flattener.writeTo(channel, fileBuilder);
  }

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Split the data into more segments (lower maxRowsPerSegment / use finer segmentGranularity) so no column exceeds Integer.MAX_VALUE rows
2. Check ingestion specs for overly large maxRowsPerSegment or maxTotalRows settings and reduce them
3. Ensure roll-up/deduplication or partitioning (e.g., hash partitioning) is used to cap per-segment row counts
4. If you truly need >2B rows per column, upgrade to a Druid version/segment format that raises the limit

Example fix

// before: one giant segment
// ingestion spec without row caps
// after
"partitionsSpec": {
  "type": "hashed",
  "maxRowsPerSegment": 5000000
}

// before writing, cap rows per segment
if (rowCount >= Integer.MAX_VALUE - 1) {
  throw new IllegalArgumentException("Segment row count would exceed column capacity; cut segment first");
}

try {
  serializer.add(value);
} catch (ColumnCapacityExceededException e) {
  // cut current segment, start a new one, retry the row there
  rollOverToNewSegment(e.getColumnName());
}

When it happens

Trigger: Calling add(double) on BlockLayoutColumnarDoublesSerializer after 2^31 rows (Integer.MAX_VALUE) have been inserted into a single column, so ++numInserted wraps to a negative value.

Common situations: Very large batch ingestion jobs or long-running streaming ingestion that pushes more than ~2.1 billion rows into a single segment without segment granularity/maxRowsPerSegment tuning; version changes that change row-count encoding.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• Column capacity exceeded
• Column capacity exceeded
• Column capacity exceeded
• Attempt to add row to swapped-out sink for segment[%s].
• ColumnCapacityExceededException

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/955cfa2c676e4d8f. Report an issue: GitHub.