What causes "Main branch snapshot changed since planning: expected but found: " in apache/iceberg?

Do not write to the main branch concurrently with this maintenance job; ensure ingestion pauses or commits are serialized with the maintenance cycle.

apache/iceberg · error · IllegalStateException

Main branch snapshot changed since planning: expected <planR

Error message

Main branch snapshot changed since planning: expected <planResult.mainSnapshotId()> but found: <mainSnapshot.snapshotId()>

What it means

Thrown by the Flink maintenance equality-delete-to-DV converter when the main table branch's current snapshot no longer matches the snapshot that was used at planning time. The operator fails fast because the committer's validateFromSnapshot would reject any DV files written against the stale plan. The next maintenance cycle will re-plan and re-index.

Source

Thrown at flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/EqualityConvertDVWriter.java:176

    super.processWatermark(mark);
  }

  private void resolveAndWrite() throws IOException {
    if (positionsByFile.isEmpty()) {
      return;
    }

    table.refresh();

    Snapshot mainSnapshot = table.snapshot(targetBranch);

    // Fail fast if the main branch changed since planning, to avoid writing DV files that the
    // committer would reject via validateFromSnapshot. The next cycle will reindex.
    if (mainSnapshot != null
        && planResult.mainSnapshotId() != null
        && mainSnapshot.snapshotId() != planResult.mainSnapshotId()) {
      throw new IllegalStateException(
          "Main branch snapshot changed since planning: expected "
              + planResult.mainSnapshotId()
              + " but found: "
              + mainSnapshot.snapshotId());
    }

    Map<String, DeleteFile> dvs = collectExistingDVs(mainSnapshot, positionsByFile.keySet());

    // Fold staging DVs into the rewrite so the writer emits one DV per data file (V3 rule). Flink
    // writes a staging DV only for a newly added data file, so it never collides with a distinct
    // existing DV: on a separate target branch collectExistingDVs has not seen it yet; on a shared
    // branch it IS that existing DV, so the put is idempotent.
    for (DeleteFile sd : planResult.stagingDVFiles()) {
      if (ContentFileUtil.isDV(sd) && sd.referencedDataFile() != null) {
        dvs.put(sd.referencedDataFile(), sd);
      }
    }

View on GitHub (pinned to 86d9c8fc54)

Solutions

Do not write to the main branch concurrently with this maintenance job; ensure ingestion pauses or commits are serialized with the maintenance cycle.
Simply re-run the maintenance cycle - the error is intentionally fail-fast and the next cycle re-plans against the new snapshot.
Use a staging branch for the rewrite so main-branch commits during processing do not invalidate the plan.
Reduce the window between planning and writing (fewer queued records, faster checkpoints) to lower race likelihood.

Example fix

// before: maintenance and ingestion both commit to main concurrently
// after: schedule the rewrite via the Iceberg rewrite API which plans and commits on a staging branch, or pause the sink during maintenance

Defensive patterns

Strategy: retry

Validate before calling

// before running/committing the cycle
Snapshot main = table.currentSnapshot();
if (!main.snapshotId().equals(planResult.mainSnapshotId())) {
  LOG.warn("Main moved since planning; restarting cycle"); // skip write, re-plan
}

Type guard

boolean planStillValid(Table table, Long plannedSnapshotId) {
  return plannedSnapshotId != null
      && table.currentSnapshot() != null
      && table.currentSnapshot().snapshotId() == plannedSnapshotId;
}

Try / catch

try {
  writer.processWatermark(watermark);
} catch (IllegalStateException e) {
  if (e.getMessage().contains("changed since planning")) {
    // expected under concurrency: restart the cycle, no data loss
    restartCycle();
  } else throw e;
}

Prevention

Do not commit to the main branch while the conversion cycle runs; serialize commits with maintenance.
Use a staging branch for rewrites so main commits cannot invalidate the plan.
Keep cycles short to shrink the planning-to-write window.
Monitor main-branch commit rates and schedule maintenance in quiet windows.

When it happens

Trigger: A concurrent commit (e.g. an ingest job or another maintenance task) advanced the main branch between the planner's snapshot capture (planResult.mainSnapshotId()) and the writer's check of the live mainSnapshot at processWatermark time.

Common situations: Running table maintenance concurrently with streaming ingestion; two maintenance jobs racing on the same table; long backpressure/watermark delays letting the main branch move before the writer fires.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/1365106dc86aed68. Report an issue: GitHub.