apache/druid · error · ColumnCapacityExceededException

Column capacity exceeded

Error message

Column capacity exceeded

What it means

Thrown by EntireLayoutColumnarDoublesSerializer.add when the number of inserted values overflows past Integer.MAX_VALUE (numInserted wraps to negative). Druid columns are int-indexed, so a serializer cannot store more than Integer.MAX_VALUE doubles; exceeding it means the segment column is too large to ever be read back correctly. ColumnCapacityExceededException signals the ingestion must fail now rather than write a corrupt segment.

Source

Thrown at processing/src/main/java/org/apache/druid/segment/data/EntireLayoutColumnarDoublesSerializer.java:78

  {
    valuesOut = segmentWriteOutMedium.makeWriteOutBytes();
  }

  @Override
  public int size()
  {
    return numInserted;
  }

  @Override
  public void add(double value) throws IOException
  {
    orderBuffer.rewind();
    orderBuffer.putDouble(value);
    valuesOut.write(orderBuffer.array());
    ++numInserted;
    if (numInserted < 0) {
      throw new ColumnCapacityExceededException(columnName);
    }
  }

  @Override
  public long getSerializedSize()
  {
    return META_SERDE_HELPER.size(this) + valuesOut.size();
  }

  @Override
  public void writeTo(WritableByteChannel channel, SegmentFileBuilder fileBuilder) throws IOException
  {
    META_SERDE_HELPER.writeTo(channel, this);
    valuesOut.writeTo(channel);
  }
}

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Partition the ingestion so each segment has fewer than Integer.MAX_VALUE rows (reduce maxRowsPerSegment / add a partition dimension)
2. If this is the in-memory incremental index, flush and persist segments earlier by lowering maxRowsInMemory
3. Split the input data into multiple tasks/partitions before ingestion
4. If you genuinely need >2B values in one column, this format cannot support it; restructure the data model

Example fix

// before (single huge segment)
maxRowsPerSegment = Integer.MAX_VALUE;
// after
maxRowsPerSegment = 5000000; // partition so no column approaches 2^31 values

if (rowsInSegment >= Integer.MAX_VALUE) {
  throw new IllegalStateException("Cannot ingest more than Integer.MAX_VALUE rows into one segment");
}

try {
  serializer.add(value);
} catch (ColumnCapacityExceededException e) {
  // abort this segment, flush what was written, and repartition into smaller segments
  throw new RuntimeException("Segment column capacity exceeded; repartition data", e);
}

When it happens

Trigger: Calling add(double) on an EntireLayoutColumnarDoublesSerializer after 2^31 values have already been inserted, i.e. writing a single double column with more than Integer.MAX_VALUE rows into one segment.

Common situations: Very large single-segment ingestion (huge maxRowsInMemory/maxBytesInMemory or a bad partitioning config) that lets one segment accumulate over 2 billion rows; ingestion of an extremely wide/deep data source without partitioning; replaying existing data into one append segment.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• Column capacity exceeded
• Column capacity exceeded
• Column capacity exceeded
• Column capacity exceeded
• Column capacity exceeded

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/e111e79022665e3f. Report an issue: GitHub.