{"record":{"id":"9b2fc7843961848d","repo":"apache/druid","slug":"too-many-values-to-store-for-s-column-try-reduci","errorCode":null,"errorMessage":"Too many values to store for %s column, try reducing maxRowsPerSegment","messagePattern":"Too many values to store for (.+?) column, try reducing maxRowsPerSegment","errorType":"exception","errorClass":"ColumnCapacityExceededException","httpStatus":null,"severity":"error","filePath":"processing/src/main/java/org/apache/druid/segment/data/VSizeColumnarMultiIntsSerializer.java","lineNumber":130,"sourceCode":"    valuesOut = segmentWriteOutMedium.makeWriteOutBytes();\n  }\n\n  @Override\n  public void addValues(IndexedInts ints) throws IOException\n  {\n    if (numBytesForMaxWritten) {\n      throw new IllegalStateException(\"written out already\");\n    }\n    for (int i = 0, size = ints.size(); i < size; i++) {\n      int value = ints.get(i);\n      Preconditions.checkState(value >= 0 && value <= maxId);\n      writeInt.write(valuesOut, value);\n    }\n    headerOut.writeInt(Ints.checkedCast(valuesOut.size()));\n\n    ++numWritten;\n    if (numWritten < 0) {\n      throw new ColumnCapacityExceededException(columnName);\n    }\n  }\n\n  @Override\n  public long getSerializedSize() throws IOException\n  {\n    writeNumBytesForMax();\n    return META_SERDE_HELPER.size(this) + headerOut.size() + valuesOut.size();\n  }\n\n  @Override\n  public void writeTo(WritableByteChannel channel, SegmentFileBuilder fileBuilder) throws IOException\n  {\n    writeNumBytesForMax();\n\n    final long numBytesWritten = headerOut.size() + valuesOut.size();\n\n    Preconditions.checkState(","sourceCodeStart":112,"sourceCodeEnd":148,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/processing/src/main/java/org/apache/druid/segment/data/VSizeColumnarMultiIntsSerializer.java#L112-L148","documentation":"VSizeColumnarMultiIntsSerializer.addValues tracks the number of rows written; when the count overflows (numWritten < 0), the column has exceeded the maximum rows a segment can hold and it throws ColumnCapacityExceededException naming the column. This guards against writing more rows than the segment format can address.","triggerScenarios":"Appending more rows to the serializer than a segment permits (~2^31 rows / configured maxRowsPerSegment limits) during ingestion, so numWritten wraps negative.","commonSituations":"Very large batch ingestion without row-count partitioning, misconfigured maxRowsPerSegment/maxTotalRows, push-down ingestion of unbounded streams without rollover.","solutions":["Lower maxRowsPerSegment (or set maxTotalRows) so ingestion partitions into more segments before overflow","Enable dynamic partitioning / intermediate handoff in the ingestion spec so segments roll over instead of growing unboundedly","Split the input batch into smaller chunks and ingest separately","Catch ColumnCapacityExceededException at the segment-creation layer to trigger a partition split"],"exampleFix":"// before: single unbounded partition\n\"partitionsSpec\": { \"type\": \"dynamic\", \"maxRowsPerSegment\": 500000000 }\n// after\n\"partitionsSpec\": { \"type\": \"dynamic\", \"maxRowsPerSegment\": 5000000, \"maxTotalRows\": 20000000 }","handlingStrategy":"try-catch","validationCode":"if (numWritten + rowsToAdd < 0 || numWritten + rowsToAdd > MAX_ROWS_PER_SEGMENT) { throw new ColumnCapacityExceededException(columnName); }","typeGuard":"boolean canAcceptRows(long currentRows, long add) { return currentRows + add > 0 && currentRows + add <= MAX_ROWS_PER_SEGMENT; }","tryCatchPattern":"try { serializer.addValues(...); } catch (ColumnCapacityExceededException e) { // roll over to a new segment partition and continue\n  handoffAndStartNewSegment(e.getColumnName()); }","preventionTips":["Set realistic maxRowsPerSegment/maxTotalRows in ingestion specs","Use dynamic partitioning with intermediate handoff for large/streaming jobs","Split oversized batches before ingestion","Monitor serialized row counts during ingestion"],"tags":["capacity","ingestion","segment-data"],"backgroundTag":"value-out-of-range","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}