{"record":{"id":"e111e79022665e3f","repo":"apache/druid","slug":"column-capacity-exceeded-e111e7","errorCode":null,"errorMessage":"Column capacity exceeded","messagePattern":"Column capacity exceeded","errorType":"exception","errorClass":"ColumnCapacityExceededException","httpStatus":null,"severity":"error","filePath":"processing/src/main/java/org/apache/druid/segment/data/EntireLayoutColumnarDoublesSerializer.java","lineNumber":78,"sourceCode":"  {\n    valuesOut = segmentWriteOutMedium.makeWriteOutBytes();\n  }\n\n  @Override\n  public int size()\n  {\n    return numInserted;\n  }\n\n  @Override\n  public void add(double value) throws IOException\n  {\n    orderBuffer.rewind();\n    orderBuffer.putDouble(value);\n    valuesOut.write(orderBuffer.array());\n    ++numInserted;\n    if (numInserted < 0) {\n      throw new ColumnCapacityExceededException(columnName);\n    }\n  }\n\n  @Override\n  public long getSerializedSize()\n  {\n    return META_SERDE_HELPER.size(this) + valuesOut.size();\n  }\n\n  @Override\n  public void writeTo(WritableByteChannel channel, SegmentFileBuilder fileBuilder) throws IOException\n  {\n    META_SERDE_HELPER.writeTo(channel, this);\n    valuesOut.writeTo(channel);\n  }\n}\n","sourceCodeStart":60,"sourceCodeEnd":95,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/processing/src/main/java/org/apache/druid/segment/data/EntireLayoutColumnarDoublesSerializer.java#L60-L95","documentation":"Thrown by EntireLayoutColumnarDoublesSerializer.add when the number of inserted values overflows past Integer.MAX_VALUE (numInserted wraps to negative). Druid columns are int-indexed, so a serializer cannot store more than Integer.MAX_VALUE doubles; exceeding it means the segment column is too large to ever be read back correctly. ColumnCapacityExceededException signals the ingestion must fail now rather than write a corrupt segment.","triggerScenarios":"Calling add(double) on an EntireLayoutColumnarDoublesSerializer after 2^31 values have already been inserted, i.e. writing a single double column with more than Integer.MAX_VALUE rows into one segment.","commonSituations":"Very large single-segment ingestion (huge maxRowsInMemory/maxBytesInMemory or a bad partitioning config) that lets one segment accumulate over 2 billion rows; ingestion of an extremely wide/deep data source without partitioning; replaying existing data into one append segment.","solutions":["Partition the ingestion so each segment has fewer than Integer.MAX_VALUE rows (reduce maxRowsPerSegment / add a partition dimension)","If this is the in-memory incremental index, flush and persist segments earlier by lowering maxRowsInMemory","Split the input data into multiple tasks/partitions before ingestion","If you genuinely need >2B values in one column, this format cannot support it; restructure the data model"],"exampleFix":"// before (single huge segment)\nmaxRowsPerSegment = Integer.MAX_VALUE;\n// after\nmaxRowsPerSegment = 5000000; // partition so no column approaches 2^31 values","handlingStrategy":"validation","validationCode":"if (rowsInSegment >= Integer.MAX_VALUE) {\n  throw new IllegalStateException(\"Cannot ingest more than Integer.MAX_VALUE rows into one segment\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  serializer.add(value);\n} catch (ColumnCapacityExceededException e) {\n  // abort this segment, flush what was written, and repartition into smaller segments\n  throw new RuntimeException(\"Segment column capacity exceeded; repartition data\", e);\n}","preventionTips":["Set maxRowsPerSegment well below 2^31 (e.g. a few million)","Use dynamic or hash partitioning so no single segment can grow unbounded","Keep maxRowsInMemory small to force frequent persists","Monitor row counts per segment during large backfills"],"tags":["capacity","segment","serialization"],"backgroundTag":"value-out-of-range","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}