apache/druid · error · ColumnCapacityExceededException

Column capacity exceeded

Error message

Column capacity exceeded

What it means

ColumnCapacityExceededException is thrown by BlockLayoutColumnarFloatsSerializer.add(float) when the number of inserted rows (numInserted, an int) overflows past Integer.MAX_VALUE. At that point the column's offset encoding cannot address further rows, so Druid aborts to avoid a corrupt segment.

Source

Thrown at processing/src/main/java/org/apache/druid/segment/data/BlockLayoutColumnarFloatsSerializer.java:104

  {
    return numInserted;
  }

  @Override
  public void add(float value) throws IOException
  {
    if (endBuffer == null) {
      throw new IllegalStateException("written out already");
    }
    if (!endBuffer.hasRemaining()) {
      endBuffer.rewind();
      flattener.write(endBuffer);
      endBuffer.clear();
    }
    endBuffer.putFloat(value);
    ++numInserted;
    if (numInserted < 0) {
      throw new ColumnCapacityExceededException(columnName);
    }
  }

  @Override
  public long getSerializedSize() throws IOException
  {
    writeEndBuffer();
    return META_SERDE_HELPER.size(this) + flattener.getSerializedSize();
  }

  @Override
  public void writeTo(WritableByteChannel channel, SegmentFileBuilder fileBuilder) throws IOException
  {
    writeEndBuffer();
    META_SERDE_HELPER.writeTo(channel, this);
    flattener.writeTo(channel, fileBuilder);
  }

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Reduce maxRowsPerSegment in the partitionsSpec so segments are cut before 2^31 rows
2. Use finer segmentGranularity or hash partitioning to distribute rows across segments
3. Enable roll-up where appropriate to reduce per-segment row counts
4. Review task failure logs for which column overflowed and adjust spec accordingly

Example fix

// before
"partitionsSpec": { "type": "dynamic", "maxRowsPerSegment": 10000000000 }
// after
"partitionsSpec": { "type": "dynamic", "maxRowsPerSegment": 5000000 }

if (numRows >= Integer.MAX_VALUE) {
  throw new IllegalArgumentException("Would exceed float column capacity; cut segment");
}

try {
  serializer.add(value);
} catch (ColumnCapacityExceededException e) {
  logger.warn("Column %s exceeded capacity; rolling segment", e.getColumnName());
  segmentRoller.roll();
}

When it happens

Trigger: Inserting more than 2^31-1 float values into a single float column of one segment via add(float).

Common situations: Oversized segments in batch ingestion (no maxRowsPerSegment), streaming tasks running far beyond intended interval/row limits, re-indexing very wide inputs without partitioning.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• Column capacity exceeded
• Column capacity exceeded
• Column capacity exceeded
• Attempt to add row to swapped-out sink for segment[%s].
• ColumnCapacityExceededException

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/b84b598e9b70771a. Report an issue: GitHub.