{"record":{"id":"9be30174059df7f4","repo":"apache/iceberg","slug":"staging-snapshot-s-on-branch-s-removes-data-fi-9be301","errorCode":null,"errorMessage":"Staging snapshot %s on branch '%s' removes data files; equality delete conversion does not support rewrites on the staging branch. Run compaction on the target branch instead.","messagePattern":"Staging snapshot (.+?) on branch '(.+?)' removes data files; equality delete conversion does not support rewrites on the staging branch\\. Run compaction on the target branch instead\\.","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/EqualityConvertPlanner.java","lineNumber":550,"sourceCode":"  /**\n   * Classifies the files added by {@code stagingSnapshot} into data files, eq delete files, and DV\n   * files. Throws if the snapshot:\n   *\n   * <ul>\n   *   <li>Removes data files (rewrites on the staging branch aren't supported).\n   *   <li>Contains V2 positional delete files (the converter expects a V3 staging branch written by\n   *       Flink, which produces only deletion vectors for deletes).\n   *   <li>Contains an eq-delete file whose {@code equalityFieldIds()} doesn't match the\n   *       builder-configured set (silent wrong-key serialization otherwise).\n   * </ul>\n   */\n  private StagingInputs retrieveStagingFiles(Snapshot stagingSnapshot) {\n    SnapshotChanges changes = SnapshotChanges.builderFor(table).snapshot(stagingSnapshot).build();\n\n    // Rewrites on the staging branch would require rewriting the corresponding DVs against new\n    // data files on target. Not implemented; fail fast instead of silently dropping work.\n    if (changes.removedDataFiles().iterator().hasNext()) {\n      throw new IllegalStateException(\n          String.format(\n              \"Staging snapshot %s on branch '%s' removes data files; \"\n                  + \"equality delete conversion does not support rewrites on the staging branch. \"\n                  + \"Run compaction on the target branch instead.\",\n              stagingSnapshot.snapshotId(), stagingBranch));\n    }\n\n    List<DataFile> newDataFiles = Lists.newArrayList();\n    List<DeleteFile> stagingDVFiles = Lists.newArrayList();\n    List<DeleteFile> eqDeleteFiles = Lists.newArrayList();\n\n    for (DataFile dataFile : changes.addedDataFiles()) {\n      newDataFiles.add(dataFile);\n    }\n\n    for (DeleteFile deleteFile : changes.addedDeleteFiles()) {\n      if (deleteFile.content() == FileContent.EQUALITY_DELETES) {\n        Set<Integer> deleteFieldIds = Sets.newHashSet(deleteFile.equalityFieldIds());","sourceCodeStart":532,"sourceCodeEnd":568,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/EqualityConvertPlanner.java#L532-L568","documentation":"EqualityConvertPlanner.retrieveStagingFiles inspects the changes of the staging-branch snapshot and rejects it outright if any data files were removed. Rewrites (compaction) on the staging branch would invalidate the DVs computed against the original data files, and this is not implemented, so the planner fails fast with an explanatory IllegalStateException instead of silently producing wrong DVs.","triggerScenarios":"inputs() -> retrieveStagingFiles finds a staging snapshot on stagingBranch whose SnapshotChanges include removedDataFiles — i.e., a compaction/rewrite commit landed on the staging branch between index generations.","commonSituations":"Misconfiguring stagingBranch to point at the branch where a regular compaction job runs; a scheduled rewrite/compaction maintenance task targeting the same branch as the equality-delete converter.","solutions":["Run compaction on the target (main) branch, not on the staging branch — as the message states.","Dedicate the staging branch exclusively to the equality-delete converter; remove other scheduled maintenance on it.","Fix the maintenance config so stagingBranch points to the correct dedicated branch.","Re-run the cycle after the staging branch contains no rewrite commits; it will reindex from a clean snapshot."],"exampleFix":"// before\nEqualityDeleteConversionConfig.builder()\n    .stagingBranch(\"main\")  // compaction also runs here\n    .targetBranch(\"main\")\n    .build();\n// after\nEqualityDeleteConversionConfig.builder()\n    .stagingBranch(\"__iceberg_edc_staging\") // dedicated branch, no rewrites\n    .targetBranch(\"main\")\n    .build();","handlingStrategy":"validation","validationCode":"// check the staging branch is rewrite-free before planning\nSnapshot s = table.snapshot(branch);\nif (s != null && SnapshotChanges.builderFor(table).snapshot(s).build()\n        .removedDataFiles().iterator().hasNext()) {\n  throw new IllegalStateException(\"Staging branch has rewrites; move compaction to target branch\");\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Dedicate the staging branch to the equality-delete converter.","Schedule compaction only on the target branch.","Audit maintenance schedules so no rewrite job targets the staging branch.","Check SnapshotChanges on the staging branch before each cycle."],"tags":["flink","planner","compaction","unsupported"],"backgroundTag":"unsupported-operation","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}