apache/druid · error · ColumnCapacityExceededException

Too many values to store for

Error message

Too many values to store for %s column, try reducing maxRowsPerSegment

What it means

VSizeColumnarMultiIntsSerializer.addValues tracks the number of rows written; when the count overflows (numWritten < 0), the column has exceeded the maximum rows a segment can hold and it throws ColumnCapacityExceededException naming the column. This guards against writing more rows than the segment format can address.

Solutions

1. Lower maxRowsPerSegment (or set maxTotalRows) so ingestion partitions into more segments before overflow
2. Enable dynamic partitioning / intermediate handoff in the ingestion spec so segments roll over instead of growing unboundedly
3. Split the input batch into smaller chunks and ingest separately
4. Catch ColumnCapacityExceededException at the segment-creation layer to trigger a partition split

Example fix

// before: single unbounded partition
"partitionsSpec": { "type": "dynamic", "maxRowsPerSegment": 500000000 }
// after
"partitionsSpec": { "type": "dynamic", "maxRowsPerSegment": 5000000, "maxTotalRows": 20000000 }

if (numWritten + rowsToAdd < 0 || numWritten + rowsToAdd > MAX_ROWS_PER_SEGMENT) { throw new ColumnCapacityExceededException(columnName); }

boolean canAcceptRows(long currentRows, long add) { return currentRows + add > 0 && currentRows + add <= MAX_ROWS_PER_SEGMENT; }

try { serializer.addValues(...); } catch (ColumnCapacityExceededException e) { // roll over to a new segment partition and continue
  handoffAndStartNewSegment(e.getColumnName()); }

When it happens

Trigger: Appending more rows to the serializer than a segment permits (~2^31 rows / configured maxRowsPerSegment limits) during ingestion, so numWritten wraps negative.

Common situations: Very large batch ingestion without row-count partitioning, misconfigured maxRowsPerSegment/maxTotalRows, push-down ingestion of unbounded streams without rollover.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• Column capacity exceeded
• Column capacity exceeded
• Column capacity exceeded
• Attempt to add row to swapped-out sink for segment
• authResult.getErrorMessage()

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/9b2fc7843961848d. Report an issue: GitHub.

Appendix: source

Thrown at processing/src/main/java/org/apache/druid/segment/data/VSizeColumnarMultiIntsSerializer.java:130

    valuesOut = segmentWriteOutMedium.makeWriteOutBytes();
  }

  @Override
  public void addValues(IndexedInts ints) throws IOException
  {
    if (numBytesForMaxWritten) {
      throw new IllegalStateException("written out already");
    }
    for (int i = 0, size = ints.size(); i < size; i++) {
      int value = ints.get(i);
      Preconditions.checkState(value >= 0 && value <= maxId);
      writeInt.write(valuesOut, value);
    }
    headerOut.writeInt(Ints.checkedCast(valuesOut.size()));

    ++numWritten;
    if (numWritten < 0) {
      throw new ColumnCapacityExceededException(columnName);
    }
  }

  @Override
  public long getSerializedSize() throws IOException
  {
    writeNumBytesForMax();
    return META_SERDE_HELPER.size(this) + headerOut.size() + valuesOut.size();
  }

  @Override
  public void writeTo(WritableByteChannel channel, SegmentFileBuilder fileBuilder) throws IOException
  {
    writeNumBytesForMax();

    final long numBytesWritten = headerOut.size() + valuesOut.size();

    Preconditions.checkState(

View on GitHub (pinned to 9b90983fd2)