apache/druid · error · ColumnCapacityExceededException

Column capacity exceeded

Error message

Column capacity exceeded

What it means

Thrown by EntireLayoutColumnarLongsSerializer.add when numInserted overflows past Integer.MAX_VALUE. Long columns are int-indexed, so no serializer instance may hold more than 2^31 - 1 values; further writes would overflow the internal int counter. ColumnCapacityExceededException fails the ingestion rather than writing a corrupt column.

Source

Thrown at processing/src/main/java/org/apache/druid/segment/data/EntireLayoutColumnarLongsSerializer.java:78

  public void open() throws IOException
  {
    valuesOut = segmentWriteOutMedium.makeWriteOutBytes();
    writer.setOutputStream(valuesOut);
  }

  @Override
  public int size()
  {
    return numInserted;
  }

  @Override
  public void add(long value) throws IOException
  {
    writer.write(value);
    ++numInserted;
    if (numInserted < 0) {
      throw new ColumnCapacityExceededException(columnName);
    }
  }

  @Override
  public long getSerializedSize() throws IOException
  {
    writer.flush();
    return META_SERDE_HELPER.size(this) + valuesOut.size();
  }

  @Override
  public void writeTo(WritableByteChannel channel, SegmentFileBuilder fileBuilder) throws IOException
  {
    writer.flush();
    META_SERDE_HELPER.writeTo(channel, this);
    valuesOut.writeTo(channel);
  }
}

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Partition the input so each segment stays below Integer.MAX_VALUE rows (tune maxRowsPerSegment, maxTotalRows)
2. Lower maxRowsInMemory to force frequent segment persists
3. Use hash/dynamic partitioning with a suitable partition dimension
4. Split oversized inputs across multiple tasks

Example fix

// before
maxRowsPerSegment = 5000000000L; // exceeds int-indexed column capacity
// after
maxRowsPerSegment = 5000000; // each segment column stays far below 2^31

if (rowsInSegment >= Integer.MAX_VALUE) {
  throw new IllegalStateException("Long column exceeds max int-indexed capacity");
}

try {
  serializer.add(value);
} catch (ColumnCapacityExceededException e) {
  throw new RuntimeException("Long column capacity exceeded; repartition data", e);
}

When it happens

Trigger: Calling add(long) on a serializer that has already accepted Integer.MAX_VALUE values; numInserted becomes negative and the check fires.

Common situations: Ingestion jobs that produce more than 2 billion rows into a single segment due to misconfigured partitioning; writing large synthetic datasets in one shot; row-limit tuning configs set to maximum.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• Column capacity exceeded
• Column capacity exceeded
• Column capacity exceeded
• Column capacity exceeded
• Column capacity exceeded

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/65ab22c2abbb1151. Report an issue: GitHub.