apache/iceberg · error · IllegalStateException

Staging snapshot %s on branch '%s' removes data files; equal

Error message

Staging snapshot %s on branch '%s' removes data files; equality delete conversion does not support rewrites on the staging branch. Run compaction on the target branch instead.

What it means

Thrown by EqualityConvertPlanner.retrieveStagingFiles when the staging-branch snapshot contains removed data files, i.e. a rewrite (compaction) happened on the staging branch. Since the converter cannot rewrite the corresponding DVs against the new data files on the target branch, it fails fast instead of silently dropping work.

Source

Thrown at flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/EqualityConvertPlanner.java:550

  /**
   * Classifies the files added by {@code stagingSnapshot} into data files, eq delete files, and DV
   * files. Throws if the snapshot:
   *
   * <ul>
   *   <li>Removes data files (rewrites on the staging branch aren't supported).
   *   <li>Contains V2 positional delete files (the converter expects a V3 staging branch written by
   *       Flink, which produces only deletion vectors for deletes).
   *   <li>Contains an eq-delete file whose {@code equalityFieldIds()} doesn't match the
   *       builder-configured set (silent wrong-key serialization otherwise).
   * </ul>
   */
  private StagingInputs retrieveStagingFiles(Snapshot stagingSnapshot) {
    SnapshotChanges changes = SnapshotChanges.builderFor(table).snapshot(stagingSnapshot).build();

    // Rewrites on the staging branch would require rewriting the corresponding DVs against new
    // data files on target. Not implemented; fail fast instead of silently dropping work.
    if (changes.removedDataFiles().iterator().hasNext()) {
      throw new IllegalStateException(
          String.format(
              "Staging snapshot %s on branch '%s' removes data files; "
                  + "equality delete conversion does not support rewrites on the staging branch. "
                  + "Run compaction on the target branch instead.",
              stagingSnapshot.snapshotId(), stagingBranch));
    }

    List<DataFile> newDataFiles = Lists.newArrayList();
    List<DeleteFile> stagingDVFiles = Lists.newArrayList();
    List<DeleteFile> eqDeleteFiles = Lists.newArrayList();

    for (DataFile dataFile : changes.addedDataFiles()) {
      newDataFiles.add(dataFile);
    }

    for (DeleteFile deleteFile : changes.addedDeleteFiles()) {
      if (deleteFile.content() == FileContent.EQUALITY_DELETES) {
        Set<Integer> deleteFieldIds = Sets.newHashSet(deleteFile.equalityFieldIds());

View on GitHub (pinned to 86d9c8fc54)

Solutions

Run compaction (rewriteDataFiles) on the target branch, not the staging branch, before or after conversion.
Point the maintenance table's staging branch config at a branch used only for staged delete commits.
Rewind or recreate the staging branch to a snapshot without data-file rewrites and rerun the conversion job.

Example fix

// before: compaction on staging branch
RewriteDataFilesAction.forTable(table).onBranch("maintenance-staging").execute();
// after: compaction on target branch
RewriteDataFilesAction.forTable(table).onBranch(targetBranch).execute();

Defensive patterns

Strategy: validation

Validate before calling

Snapshot staging = table.snapshotForBranch(stagingBranch);
SnapshotChanges changes = SnapshotChanges.builderFor(table).snapshot(staging).build();
if (changes.removedDataFiles().iterator().hasNext()) {
  throw new IllegalStateException("Staging branch has rewrites; run compaction on target branch instead");
}
// else safe to run conversion

Try / catch

try {
  runEqualityConvertJob(table, cfg);
} catch (IllegalStateException e) {
  if (e.getMessage().contains("removes data files")) {
    LOG.warn("Staging branch has compaction rewrites; redirect compaction to target branch", e);
  } else throw e;
}

Prevention

Only use the staging branch for staged delete commits
Schedule compaction (rewriteDataFiles) on the target branch
Review any automated rewrite policies for branch targeting
Validate the staging snapshot before conversion with SnapshotChanges

When it happens

Trigger: A snapshot staged on stagingBranch was produced by a rewriteDataFiles/compaction job (or any operation that removes data files), and then the equality-delete-conversion planner inspects that snapshot via SnapshotChanges.removedDataFiles().

Common situations: Scheduling compaction on the maintenance staging branch by mistake; sharing one branch for both compaction and delete conversion in the maintenance configuration; an automated rewrite policy targeting the wrong branch.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/bb22cdb8668b1cf6. Report an issue: GitHub.