apache/iceberg · error · IllegalStateException

Staging snapshot %s on branch '%s' removes data files; equal

Error message

Staging snapshot %s on branch '%s' removes data files; equality delete conversion does not support rewrites on the staging branch. Run compaction on the target branch instead.

What it means

EqualityConvertPlanner.retrieveStagingFiles inspects the changes of the staging-branch snapshot and rejects it outright if any data files were removed. Rewrites (compaction) on the staging branch would invalidate the DVs computed against the original data files, and this is not implemented, so the planner fails fast with an explanatory IllegalStateException instead of silently producing wrong DVs.

Source

Thrown at flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/EqualityConvertPlanner.java:550

  /**
   * Classifies the files added by {@code stagingSnapshot} into data files, eq delete files, and DV
   * files. Throws if the snapshot:
   *
   * <ul>
   *   <li>Removes data files (rewrites on the staging branch aren't supported).
   *   <li>Contains V2 positional delete files (the converter expects a V3 staging branch written by
   *       Flink, which produces only deletion vectors for deletes).
   *   <li>Contains an eq-delete file whose {@code equalityFieldIds()} doesn't match the
   *       builder-configured set (silent wrong-key serialization otherwise).
   * </ul>
   */
  private StagingInputs retrieveStagingFiles(Snapshot stagingSnapshot) {
    SnapshotChanges changes = SnapshotChanges.builderFor(table).snapshot(stagingSnapshot).build();

    // Rewrites on the staging branch would require rewriting the corresponding DVs against new
    // data files on target. Not implemented; fail fast instead of silently dropping work.
    if (changes.removedDataFiles().iterator().hasNext()) {
      throw new IllegalStateException(
          String.format(
              "Staging snapshot %s on branch '%s' removes data files; "
                  + "equality delete conversion does not support rewrites on the staging branch. "
                  + "Run compaction on the target branch instead.",
              stagingSnapshot.snapshotId(), stagingBranch));
    }

    List<DataFile> newDataFiles = Lists.newArrayList();
    List<DeleteFile> stagingDVFiles = Lists.newArrayList();
    List<DeleteFile> eqDeleteFiles = Lists.newArrayList();

    for (DataFile dataFile : changes.addedDataFiles()) {
      newDataFiles.add(dataFile);
    }

    for (DeleteFile deleteFile : changes.addedDeleteFiles()) {
      if (deleteFile.content() == FileContent.EQUALITY_DELETES) {
        Set<Integer> deleteFieldIds = Sets.newHashSet(deleteFile.equalityFieldIds());

View on GitHub (pinned to 86d9c8fc54)

Solutions

Run compaction on the target (main) branch, not on the staging branch — as the message states.
Dedicate the staging branch exclusively to the equality-delete converter; remove other scheduled maintenance on it.
Fix the maintenance config so stagingBranch points to the correct dedicated branch.
Re-run the cycle after the staging branch contains no rewrite commits; it will reindex from a clean snapshot.

Example fix

// before
EqualityDeleteConversionConfig.builder()
    .stagingBranch("main")  // compaction also runs here
    .targetBranch("main")
    .build();
// after
EqualityDeleteConversionConfig.builder()
    .stagingBranch("__iceberg_edc_staging") // dedicated branch, no rewrites
    .targetBranch("main")
    .build();

Defensive patterns

Strategy: validation

Validate before calling

// check the staging branch is rewrite-free before planning
Snapshot s = table.snapshot(branch);
if (s != null && SnapshotChanges.builderFor(table).snapshot(s).build()
        .removedDataFiles().iterator().hasNext()) {
  throw new IllegalStateException("Staging branch has rewrites; move compaction to target branch");
}

Prevention

Dedicate the staging branch to the equality-delete converter.
Schedule compaction only on the target branch.
Audit maintenance schedules so no rewrite job targets the staging branch.
Check SnapshotChanges on the staging branch before each cycle.

When it happens

Trigger: inputs() -> retrieveStagingFiles finds a staging snapshot on stagingBranch whose SnapshotChanges include removedDataFiles — i.e., a compaction/rewrite commit landed on the staging branch between index generations.

Common situations: Misconfiguring stagingBranch to point at the branch where a regular compaction job runs; a scheduled rewrite/compaction maintenance task targeting the same branch as the equality-delete converter.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/9be30174059df7f4. Report an issue: GitHub.