apache/druid · error · ColumnCapacityExceededException

Column capacity exceeded

Error message

Column capacity exceeded

What it means

ColumnCapacityExceededException from BlockLayoutColumnarLongsSerializer.add(long) fires when numInserted overflows Integer.MAX_VALUE. Long columns are row-addressed by int offsets, so beyond 2^31-1 rows Druid cannot encode more values and aborts segment creation to prevent corruption.

Source

Thrown at processing/src/main/java/org/apache/druid/segment/data/BlockLayoutColumnarLongsSerializer.java:116

  @Override
  public void add(long value) throws IOException
  {
    if (endBuffer == null) {
      throw new IllegalStateException("written out already");
    }
    if (numInserted == numInsertedForNextFlush) {
      numInsertedForNextFlush += sizePer;
      writer.flush();
      endBuffer.flip();
      flattener.write(endBuffer);
      endBuffer.clear();
      writer.setBuffer(endBuffer);
    }

    writer.write(value);
    ++numInserted;
    if (numInserted < 0) {
      throw new ColumnCapacityExceededException(columnName);
    }
  }

  @Override
  public long getSerializedSize() throws IOException
  {
    writeEndBuffer();
    return META_SERDE_HELPER.size(this) + flattener.getSerializedSize();
  }

  @Override
  public void writeTo(WritableByteChannel channel, SegmentFileBuilder fileBuilder) throws IOException
  {
    writeEndBuffer();
    META_SERDE_HELPER.writeTo(channel, this);
    flattener.writeTo(channel, fileBuilder);
  }

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Set maxRowsPerSegment well below Integer.MAX_VALUE in partitionsSpec
2. Partition by time or hash so no single segment exceeds ~2.1B rows
3. Use roll-up/aggregation to shrink row counts
4. Upgrade Druid if a newer segment format lifts the limit

Example fix

// before
// unbounded dynamic partitioning
"partitionsSpec": { "type": "dynamic" }
// after
"partitionsSpec": { "type": "dynamic", "maxRowsPerSegment": 5000000 }

if (numRows >= Integer.MAX_VALUE) {
  throw new IllegalArgumentException("Would exceed long column capacity; cut segment");
}

try {
  serializer.add(value);
} catch (ColumnCapacityExceededException e) {
  // cut the segment and restart ingestion at the failed row
  cutSegmentAndRetry(e.getColumnName());
}

When it happens

Trigger: Inserting more than 2^31-1 rows into a single long-typed column of one segment via add(long).

Common situations: Very high-volume ingestion without row-count partitioning; metric columns in billion-row segments; task retries that keep appending to the same serializer.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• Column capacity exceeded
• Column capacity exceeded
• Column capacity exceeded
• Attempt to add row to swapped-out sink for segment[%s].
• ColumnCapacityExceededException

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/36ecc27b5a8bf79b. Report an issue: GitHub.