apache/beam · error · IllegalStateException

Position delete index cardinality exceeds Integer.MAX_VALUE

Error message

Position delete index cardinality exceeds Integer.MAX_VALUE: {}

What it means

CdcReadUtils.sortedDeletePositions materializes a PositionDeleteIndex into a sorted long[] of delete positions for binary-search lookup. Because the array length must be an int, a delete index whose cardinality exceeds Integer.MAX_VALUE cannot be materialized, and the method throws IllegalStateException with the offending cardinality.

Solutions

1. Reduce the number of position deletes (run Iceberg rewrite/delete compaction / expire delete files) so no single data file has >2^31 deletes.
2. Split the read so deletes are applied in smaller batches (smaller data files / frequent compaction).
3. Verify the cardinality value — an unexpectedly huge number may indicate corrupt delete-file metadata; validate table metadata.
4. Long-term: patch the reader to stream deletes instead of materializing an int-sized array.

Example fix

// before
spark.sql("CALL catalog.system.rewrite_data_files(table => 'db.tbl')") // never run; deletes accumulate
// after
-- periodically compact so delete cardinality per file stays small
spark.sql("CALL catalog.system.rewrite_position_delete_files(table => 'db.tbl', options => map('rewrite-all','true'))");

// pre-flight: bound delete cardinality per data file via table metadata/compaction policy
long card = posIndex.cardinality();
if (card > Integer.MAX_VALUE) {
  throw new IllegalStateException("Compact table: delete cardinality " + card + " exceeds int range");
}

if (posIndex.cardinality() > Integer.MAX_VALUE) return null; // fall back instead of materializing

try {
  long[] positions = sortedDeletePositions(posIndex);
} catch (IllegalStateException e) {
  if (e.getMessage().contains("Integer.MAX_VALUE")) {
    // route file through a streaming delete filter instead
  } else { throw e; }
}

When it happens

Trigger: A position-delete index accumulated across delete files for a data file contains more than 2,147,483,647 (2^31-1) deleted positions when sortedDeletePositions is called in the CDC read path.

Common situations: Extremely large single data files with an enormous number of position deletes — practically a pathological/outlier table or a runaway deletion process; usually only reachable with multi-terabyte files and billions of deletes.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• Expected at least one overlapping task in bidirectional list
• Invalid starting strategy. Valid values are
• the following options are currently only available when…
• Unexpected Beam type
• Unexpected logical type

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c4e95e97b2a1dad5. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/CdcReadUtils.java:455

        TypeUtil.join(existingDeletesFilter.requiredSchema(), addedDeletesReader.requiredSchema());
    CloseableIterable<Record> records =
        createReader(
            task,
            table,
            scanConfig,
            requiredSchema,
            Expressions.alwaysTrue(),
            readStart,
            readEnd - readStart);
    CloseableIterable<Record> liveRecords = existingDeletesFilter.filter(records);
    return PositionPushdownResult.of(addedDeletesReader.read(liveRecords), preloadedDeletes);
  }

  /** Materializes a sorted long[] of the positions in {@code posIndex} for binary-search lookup. */
  private static long[] sortedDeletePositions(PositionDeleteIndex posIndex) {
    long cardinality = posIndex.cardinality();
    if (cardinality > Integer.MAX_VALUE) {
      throw new IllegalStateException(
          "Position delete index cardinality exceeds Integer.MAX_VALUE: " + cardinality);
    }
    long[] arr = new long[(int) cardinality];
    int[] idx = {0};
    posIndex.forEach(p -> arr[idx[0]++] = p);
    // forEach is ordered for the bitmap-backed implementation, but the interface doesn't
    // promise it, so sort defensively. Cheap relative to the I/O it gates.
    Arrays.sort(arr);
    return arr;
  }

  /** Returns true iff {@code sortedDeletes} contains any value in {@code [start, end)}. */
  private static boolean anyInRange(long[] sortedDeletes, long startInclusive, long endExclusive) {
    if (sortedDeletes.length == 0) {
      return false;
    }
    int i = Arrays.binarySearch(sortedDeletes, startInclusive);
    if (i < 0) {

View on GitHub (pinned to 12126d8942)