apache/druid · critical · ColumnCapacityExceededException

ColumnCapacityExceededException

Error message

ColumnCapacityExceededException

What it means

During column serialization, each call to serialize() increments rowCount; if rowCount overflows past Integer.MAX_VALUE (rowCount becomes negative), the writer throws ColumnCapacityExceededException because the column cannot hold more rows than an int can count.

Source

Thrown at extensions-contrib/spectator-histogram/src/main/java/org/apache/druid/spectator/histogram/SpectatorHistogramSerializer.java:87

  {
    this.columnName = columnName;
    this.segmentWriteOutMedium = segmentWriteOutMedium;
    this.objectStrategy = strategy;
  }

  @Override
  public void open() throws IOException
  {
    this.offsetsHeader = NullableOffsetsHeader.create(segmentWriteOutMedium);
    this.valuesBuffer = segmentWriteOutMedium.makeWriteOutBytes();
  }

  @Override
  public void serialize(ColumnValueSelector<?> selector) throws IOException
  {
    rowCount++;
    if (rowCount < 0) {
      throw new ColumnCapacityExceededException(columnName);
    }
    Object value = selector.getObject();
    if (value == null) {
      offsetsHeader.writeNull();
    } else {
      objectStrategy.writeTo((SpectatorHistogram) value, valuesBuffer);
      offsetsHeader.writeOffset(Ints.checkedCast(valuesBuffer.size()));
    }
  }

  @Override
  public long getSerializedSize()
  {
    // Meta header, Offsets, Values
    return META_SERDE_HELPER.size(this) + offsetsHeader.getSerializedSize() + valuesBuffer.size();
  }

  @Override

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Reduce rows per segment: lower druid.segment.rowNumTarget / maxNumSegmentPartitions or tune partitionsSpec so segments stay well below Integer.MAX_VALUE rows.
2. Re-partition the ingestion task input so no single task writes >2^31 rows.
3. If thrown, split the input data and run separate ingestion tasks.
4. Verify no counter bug is causing repeated serialize calls for the same rows.

Example fix

// before (tuning)
// single task ingesting unbounded rows
// after
dynamic partitionsSpec with maxTotalRows well below 2_000_000_000, or ranged partitions on time

if (taskMaxRows > Integer.MAX_VALUE - 1000) { throw new IllegalArgumentException("segment row budget too close to Integer.MAX_VALUE; reduce partitions"); }

try { serializer.serialize(selector); } catch (ColumnCapacityExceededException e) { throw new IOException("segment exceeded row capacity; re-partition input: " + e.getMessage(), e); }

When it happens

Trigger: Ingesting more than ~2.1 billion rows into a single spectator-histogram column segment; rowCount wraps to negative on the 2^31st row and serialize() throws.

Common situations: Very large batch tasks without partitioning; misconfigured maxRowNumInSegment allowing unbounded segment growth; ingestion of an entire huge table into one segment.

Understand the failure class

Background: "File too large" / "file size exceeds limit" errors: why libraries cap file sizes and how to fix them — this error's family across 46 libraries.

Related errors

• Pushed segments[%s] are different from the requested ones[%s
• Can't find segmentsForSequence for sequence[%s]
• Can't find pushedSegments for segment[%s]
• Cannot deserialize type[%s] to an RoaringBitmap64Counter:
• Index[%d] >= size[%d]

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/c75ca7e384ecf4d8. Report an issue: GitHub.