apache/iceberg · error · IllegalStateException

Staging snapshot %s on branch '%s' removes data files; equal

Error message

Staging snapshot %s on branch '%s' removes data files; equality delete conversion does not support rewrites on the staging branch. Run compaction on the target branch instead.

What it means

The equality-delete conversion planner detected that the staging-branch snapshot removed (rewrote away) data files. Compaction-style rewrites on the staging branch would require rewriting the corresponding DVs against new data files on the target, which is not implemented, so the planner fails fast rather than silently dropping those deletes.

Source

Thrown at flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/EqualityConvertPlanner.java:550

  /**
   * Classifies the files added by {@code stagingSnapshot} into data files, eq delete files, and DV
   * files. Throws if the snapshot:
   *
   * <ul>
   *   <li>Removes data files (rewrites on the staging branch aren't supported).
   *   <li>Contains V2 positional delete files (the converter expects a V3 staging branch written by
   *       Flink, which produces only deletion vectors for deletes).
   *   <li>Contains an eq-delete file whose {@code equalityFieldIds()} doesn't match the
   *       builder-configured set (silent wrong-key serialization otherwise).
   * </ul>
   */
  private StagingInputs retrieveStagingFiles(Snapshot stagingSnapshot) {
    SnapshotChanges changes = SnapshotChanges.builderFor(table).snapshot(stagingSnapshot).build();

    // Rewrites on the staging branch would require rewriting the corresponding DVs against new
    // data files on target. Not implemented; fail fast instead of silently dropping work.
    if (changes.removedDataFiles().iterator().hasNext()) {
      throw new IllegalStateException(
          String.format(
              "Staging snapshot %s on branch '%s' removes data files; "
                  + "equality delete conversion does not support rewrites on the staging branch. "
                  + "Run compaction on the target branch instead.",
              stagingSnapshot.snapshotId(), stagingBranch));
    }

    List<DataFile> newDataFiles = Lists.newArrayList();
    List<DeleteFile> stagingDVFiles = Lists.newArrayList();
    List<DeleteFile> eqDeleteFiles = Lists.newArrayList();

    for (DataFile dataFile : changes.addedDataFiles()) {
      newDataFiles.add(dataFile);
    }

    for (DeleteFile deleteFile : changes.addedDeleteFiles()) {
      if (deleteFile.content() == FileContent.EQUALITY_DELETES) {
        Set<Integer> deleteFieldIds = Sets.newHashSet(deleteFile.equalityFieldIds());

View on GitHub (pinned to 86d9c8fc54)

Solutions

Run compaction on the target branch, not the staging branch, as the message advises.
Keep the staging branch reserved for DV-writing rewrites only (equality delete conversion commits).
Re-create/rebase the staging branch from a snapshot that contains no rewrite commits and re-run the conversion.
If compaction on staging is required in your pipeline, disable it or partition it out from the conversion branch.

Example fix

// before
SparkActions.get(spark).rewriteDataFiles(table).onBranch("staging").execute();
// after
SparkActions.get(spark).rewriteDataFiles(table).onBranch("main").execute();

Defensive patterns

Strategy: validation

Validate before calling

// before running conversion, assert the staging branch has no rewrite commits
Snapshot staging = table.snapshot(branch);
boolean hasRewrites = SnapshotChanges.builderFor(table).snapshot(staging).build()
    .removedDataFiles().iterator().hasNext();
if (hasRewrites) throw new IllegalStateException("Compaction detected on staging branch");

Type guard

boolean stagingFreeOfRewrites(Table table, String branch) {
  Snapshot s = table.snapshot(branch);
  return s == null || !SnapshotChanges.builderFor(table).snapshot(s).build()
      .removedDataFiles().iterator().hasNext();
}

Try / catch

try {
  planner.inputs();
} catch (IllegalStateException e) {
  if (e.getMessage().contains("does not support rewrites")) {
    moveCompactionToTargetBranch(); rerun();
  } else throw e;
}

Prevention

Configure compaction jobs to run on the target/main branch only.
Keep the staging branch dedicated to DV-writing maintenance commits.
Audit branch-level scheduled jobs (rewrite_data_files targets) before enabling conversion.
Gate conversion on a pre-flight check of SnapshotChanges for the staging snapshot.

When it happens

Trigger: A snapshot on the staging branch was produced by a rewrite/compaction (replace/delete of data files) instead of only appends plus DV writes, and EqualityConvertPlanner.inputs() -> retrieveStagingFiles() inspects SnapshotChanges.removedDataFiles().

Common situations: Someone ran compaction (rewrite_data_files) on the staging branch; a maintenance configuration points the planner at a branch that receives regular compaction; copied snapshots include rewrite commits.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/07c5f8196c4d1513. Report an issue: GitHub.