{"record":{"id":"c4e95e97b2a1dad5","repo":"apache/beam","slug":"position-delete-index-cardinality-exceeds-integer-max-value","errorCode":null,"errorMessage":"Position delete index cardinality exceeds Integer.MAX_VALUE: {}","messagePattern":"Position delete index cardinality exceeds Integer\\.MAX_VALUE: (.+?)","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/CdcReadUtils.java","lineNumber":455,"sourceCode":"        TypeUtil.join(existingDeletesFilter.requiredSchema(), addedDeletesReader.requiredSchema());\n    CloseableIterable<Record> records =\n        createReader(\n            task,\n            table,\n            scanConfig,\n            requiredSchema,\n            Expressions.alwaysTrue(),\n            readStart,\n            readEnd - readStart);\n    CloseableIterable<Record> liveRecords = existingDeletesFilter.filter(records);\n    return PositionPushdownResult.of(addedDeletesReader.read(liveRecords), preloadedDeletes);\n  }\n\n  /** Materializes a sorted long[] of the positions in {@code posIndex} for binary-search lookup. */\n  private static long[] sortedDeletePositions(PositionDeleteIndex posIndex) {\n    long cardinality = posIndex.cardinality();\n    if (cardinality > Integer.MAX_VALUE) {\n      throw new IllegalStateException(\n          \"Position delete index cardinality exceeds Integer.MAX_VALUE: \" + cardinality);\n    }\n    long[] arr = new long[(int) cardinality];\n    int[] idx = {0};\n    posIndex.forEach(p -> arr[idx[0]++] = p);\n    // forEach is ordered for the bitmap-backed implementation, but the interface doesn't\n    // promise it, so sort defensively. Cheap relative to the I/O it gates.\n    Arrays.sort(arr);\n    return arr;\n  }\n\n  /** Returns true iff {@code sortedDeletes} contains any value in {@code [start, end)}. */\n  private static boolean anyInRange(long[] sortedDeletes, long startInclusive, long endExclusive) {\n    if (sortedDeletes.length == 0) {\n      return false;\n    }\n    int i = Arrays.binarySearch(sortedDeletes, startInclusive);\n    if (i < 0) {","sourceCodeStart":437,"sourceCodeEnd":473,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/CdcReadUtils.java#L437-L473","documentation":"CdcReadUtils.sortedDeletePositions materializes a PositionDeleteIndex into a sorted long[] of delete positions for binary-search lookup. Because the array length must be an int, a delete index whose cardinality exceeds Integer.MAX_VALUE cannot be materialized, and the method throws IllegalStateException with the offending cardinality.","triggerScenarios":"A position-delete index accumulated across delete files for a data file contains more than 2,147,483,647 (2^31-1) deleted positions when sortedDeletePositions is called in the CDC read path.","commonSituations":"Extremely large single data files with an enormous number of position deletes — practically a pathological/outlier table or a runaway deletion process; usually only reachable with multi-terabyte files and billions of deletes.","solutions":["Reduce the number of position deletes (run Iceberg rewrite/delete compaction / expire delete files) so no single data file has >2^31 deletes.","Split the read so deletes are applied in smaller batches (smaller data files / frequent compaction).","Verify the cardinality value — an unexpectedly huge number may indicate corrupt delete-file metadata; validate table metadata.","Long-term: patch the reader to stream deletes instead of materializing an int-sized array."],"exampleFix":"// before\nspark.sql(\"CALL catalog.system.rewrite_data_files(table => 'db.tbl')\") // never run; deletes accumulate\n// after\n-- periodically compact so delete cardinality per file stays small\nspark.sql(\"CALL catalog.system.rewrite_position_delete_files(table => 'db.tbl', options => map('rewrite-all','true'))\");","handlingStrategy":"validation","validationCode":"// pre-flight: bound delete cardinality per data file via table metadata/compaction policy\nlong card = posIndex.cardinality();\nif (card > Integer.MAX_VALUE) {\n  throw new IllegalStateException(\"Compact table: delete cardinality \" + card + \" exceeds int range\");\n}","typeGuard":"if (posIndex.cardinality() > Integer.MAX_VALUE) return null; // fall back instead of materializing","tryCatchPattern":"try {\n  long[] positions = sortedDeletePositions(posIndex);\n} catch (IllegalStateException e) {\n  if (e.getMessage().contains(\"Integer.MAX_VALUE\")) {\n    // route file through a streaming delete filter instead\n  } else { throw e; }\n}","preventionTips":["Run regular position-delete compaction (rewrite_position_delete_files).","Keep data files modestly sized so per-file delete counts stay well below 2^31.","Investigate abnormally high cardinality — may signal corrupt delete metadata."],"tags":["java","iceberg","cdc","overflow","delete-files"],"backgroundTag":"value-out-of-range","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}