apache/iceberg · error · IllegalStateException

Main branch snapshot changed since planning: expected {} but

Error message

Main branch snapshot changed since planning: expected {} but found: {}

What it means

EqualityConvertDVWriter.resolveAndWrite() fails fast when the main branch snapshot observed at runtime differs from the snapshot id the DV plan was built against (planResult.mainSnapshotId()). Writing deletion vectors against a stale plan could produce files the committer's validateFromSnapshot would reject, so the writer aborts and the next cycle reindexes.

Source

Thrown at flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/EqualityConvertDVWriter.java:176

    super.processWatermark(mark);
  }

  private void resolveAndWrite() throws IOException {
    if (positionsByFile.isEmpty()) {
      return;
    }

    table.refresh();

    Snapshot mainSnapshot = table.snapshot(targetBranch);

    // Fail fast if the main branch changed since planning, to avoid writing DV files that the
    // committer would reject via validateFromSnapshot. The next cycle will reindex.
    if (mainSnapshot != null
        && planResult.mainSnapshotId() != null
        && mainSnapshot.snapshotId() != planResult.mainSnapshotId()) {
      throw new IllegalStateException(
          "Main branch snapshot changed since planning: expected "
              + planResult.mainSnapshotId()
              + " but found: "
              + mainSnapshot.snapshotId());
    }

    Map<String, DeleteFile> dvs = collectExistingDVs(mainSnapshot, positionsByFile.keySet());

    // Fold staging DVs into the rewrite so the writer emits one DV per data file (V3 rule). Flink
    // writes a staging DV only for a newly added data file, so it never collides with a distinct
    // existing DV: on a separate target branch collectExistingDVs has not seen it yet; on a shared
    // branch it IS that existing DV, so the put is idempotent.
    for (DeleteFile sd : planResult.stagingDVFiles()) {
      if (ContentFileUtil.isDV(sd) && sd.referencedDataFile() != null) {
        dvs.put(sd.referencedDataFile(), sd);
      }
    }

View on GitHub (pinned to 86d9c8fc54)

Solutions

Don't write to the main table concurrently with the conversion job, or run conversion on an isolated branch schedule
Let the next maintenance cycle reindex — this error is designed to be recovered automatically
Reduce the window by triggering processing sooner after planning (commit-triggered rather than watermark-delayed)
Check the table's commit history to identify the competing writer

Defensive patterns

Strategy: validation

Validate before calling

// before running conversion, ensure no concurrent commits to main branch
Snapshot current = table.snapshot(mainBranch);
if (plannedMainSnapshotId != current.snapshotId()) {
  // re-plan the cycle before writing DVs
}

Prevention

Avoid concurrent writers to the main branch during conversion cycles
Schedule conversion during write-quiescent windows
Trigger processing immediately after planning to shrink the race window
Design the task to tolerate the automatic reindex on the next cycle

When it happens

Trigger: Between scan planning and watermark-triggered write, another commit advanced the main branch (e.g. a concurrent writer or a prior maintenance cycle committed), so mainSnapshot.snapshotId() != planResult.mainSnapshotId().

Common situations: Concurrent stream/batch writers committing to the same table during equality-delete conversion; overlapping maintenance jobs; manual Spark/Flink commits landing mid-cycle.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/342b6143c29409bf. Report an issue: GitHub.