apache/iceberg · error · IllegalStateException

Staging snapshot %s on branch '%s' removes data files; equal

Error message

Staging snapshot %s on branch '%s' removes data files; equality delete conversion does not support rewrites on the staging branch. Run compaction on the target branch instead.

What it means

EqualityConvertPlanner.retrieveStagingFiles() inspects the staged snapshot's changes and rejects snapshots that removed (rewrote) data files. Equality delete conversion cannot rewrite the corresponding DVs against new data files, so it fails fast instead of silently dropping work, advising compaction be run on the target branch instead.

Source

Thrown at flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/EqualityConvertPlanner.java:550

  /**
   * Classifies the files added by {@code stagingSnapshot} into data files, eq delete files, and DV
   * files. Throws if the snapshot:
   *
   * <ul>
   *   <li>Removes data files (rewrites on the staging branch aren't supported).
   *   <li>Contains V2 positional delete files (the converter expects a V3 staging branch written by
   *       Flink, which produces only deletion vectors for deletes).
   *   <li>Contains an eq-delete file whose {@code equalityFieldIds()} doesn't match the
   *       builder-configured set (silent wrong-key serialization otherwise).
   * </ul>
   */
  private StagingInputs retrieveStagingFiles(Snapshot stagingSnapshot) {
    SnapshotChanges changes = SnapshotChanges.builderFor(table).snapshot(stagingSnapshot).build();

    // Rewrites on the staging branch would require rewriting the corresponding DVs against new
    // data files on target. Not implemented; fail fast instead of silently dropping work.
    if (changes.removedDataFiles().iterator().hasNext()) {
      throw new IllegalStateException(
          String.format(
              "Staging snapshot %s on branch '%s' removes data files; "
                  + "equality delete conversion does not support rewrites on the staging branch. "
                  + "Run compaction on the target branch instead.",
              stagingSnapshot.snapshotId(), stagingBranch));
    }

    List<DataFile> newDataFiles = Lists.newArrayList();
    List<DeleteFile> stagingDVFiles = Lists.newArrayList();
    List<DeleteFile> eqDeleteFiles = Lists.newArrayList();

    for (DataFile dataFile : changes.addedDataFiles()) {
      newDataFiles.add(dataFile);
    }

    for (DeleteFile deleteFile : changes.addedDeleteFiles()) {
      if (deleteFile.content() == FileContent.EQUALITY_DELETES) {
        Set<Integer> deleteFieldIds = Sets.newHashSet(deleteFile.equalityFieldIds());

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Run compaction/rewrite jobs on the target (main) branch, not the staging branch
2. Rewind or reset the staging branch to a snapshot before the rewrite, then re-run conversion
3. Start a fresh staging branch for the conversion cycle
4. Adjust any automation so branch-targeted optimization excludes the staging branch

Example fix

// before
CALL iceberg.system.rewrite_data_files(table => 'db.t', branch => 'staging')
// after
CALL iceberg.system.rewrite_data_files(table => 'db.t', branch => 'main')

// before running conversion, check staging snapshot for rewrites
SnapshotChanges changes = SnapshotChanges.builderFor(table).snapshot(stagingSnapshot).build();
boolean hasRewrites = changes.removedDataFiles().iterator().hasNext();
if (hasRewrites) { /* reset staging branch or run compaction on main */ }

When it happens

Trigger: The staging branch's latest snapshot contains a rewrite/replace commit (removedDataFiles non-empty) — e.g. compaction, file-rewrite, or expire/rewrite actions were run on the staging branch between conversion cycles.

Common situations: A separate compaction or rewrite job targeting the staging branch; user mistake running maintenance (rewrite_data_files) against the staging branch instead of main; automated optimization service rewriting small files on the wrong branch.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• Main branch snapshot changed since planning: expected {} but
• Staging snapshot %s on branch '%s' contains a V2 positional
• this.getClass().getName() + " doesn't implement removedDelet
• Bitmap decoding has not been implemented
• Altering schema is not supported in the old alterTable API.

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/7332150733459401. Report an issue: GitHub.