apache/iceberg · error · RuntimeException

Cannot commit rewrite because of a ValidationException or Co

Error message

Cannot commit rewrite because of a ValidationException or CommitFailedException. This usually means that this rewrite has conflicted with another concurrent Iceberg operation. To reduce the likelihood of conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s. Separate smaller rewrite commits can succeed independently while any commits that conflict with another Iceberg operation will be ignored. This mode will create additional snapshots in the table history, one for each commit.

What it means

In RewriteDataFilesSparkAction.doExecute, a group rewrite failed with ValidationException or CommitFailedException on its single commit. The action wraps it in a RuntimeException explaining that the rewrite conflicted with a concurrent Iceberg operation, and suggests enabling partial progress mode so smaller commits can succeed independently.

Source

Thrown at spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java:315

          .run(commitManager::abortFileGroup);
      throw e;
    } finally {
      rewriteService.shutdown();
    }

    try {
      commitManager.commitOrClean(Sets.newHashSet(rewrittenGroups));
    } catch (ValidationException | CommitFailedException e) {
      String errorMessage =
          String.format(
              "Cannot commit rewrite because of a ValidationException or CommitFailedException. This usually means that "
                  + "this rewrite has conflicted with another concurrent Iceberg operation. To reduce the likelihood of "
                  + "conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s. "
                  + "Separate smaller rewrite commits can succeed independently while any commits that conflict with "
                  + "another Iceberg operation will be ignored. This mode will create additional snapshots in the table "
                  + "history, one for each commit.",
              PARTIAL_PROGRESS_ENABLED, PARTIAL_PROGRESS_MAX_COMMITS);
      throw new RuntimeException(errorMessage, e);
    }

    List<FileGroupRewriteResult> rewriteResults =
        rewrittenGroups.stream().map(RewriteFileGroup::asResult).collect(Collectors.toList());
    return ImmutableRewriteDataFiles.Result.builder().rewriteResults(rewriteResults);
  }

  private Builder doExecuteWithPartialProgress(
      FileRewritePlan<FileGroupInfo, FileScanTask, DataFile, RewriteFileGroup> plan,
      RewriteDataFilesCommitManager commitManager) {
    ExecutorService rewriteService = rewriteService();

    // start commit service
    int groupsPerCommit = IntMath.divide(plan.totalGroupCount(), maxCommits, RoundingMode.CEILING);
    RewriteDataFilesCommitManager.CommitService commitService =
        commitManager.service(groupsPerCommit);
    commitService.start();

View on GitHub (pinned to 86d9c8fc54)

Solutions

Re-run the rewrite when concurrent writers are paused or less active.
Enable partial progress: .option("partial-progress.enabled", "true") so the rewrite is split into multiple smaller commits via partial-progress.max-commits.
Retry the action — CommitFailedException conflicts may pass on a quieter table.
Investigate the chained exception (ValidationException details) if the conflict is caused by a deterministic requirement violation rather than concurrency.

Example fix

// before
SparkActions.get(spark).rewriteDataFiles(table).execute();
// after
SparkActions.get(spark).rewriteDataFiles(table)
    .option("partial-progress.enabled", "true")
    .option("partial-progress.max-commits", "10")
    .execute();

Defensive patterns

Strategy: retry

Validate before calling

// Check for concurrent commits before rewriting
Snapshot current = table.currentSnapshot();
// Ensure no other job is committing: monitor table history frequency
System.out.println("Current snapshot: " + current.snapshotId());

Try / catch

try {
  rewrite.execute();
} catch (RuntimeException e) {
  if (e.getCause() instanceof ValidationException || e.getCause() instanceof CommitFailedException) {
    // retry later or enable partial-progress mode
  } else throw e;
}

Prevention

Enable partial-progress.enabled=true for tables with steady concurrent writes.
Schedule compaction in windows with low ingestion traffic.
Retry once on conflict failures before escalating — single-commit conflicts are transient.

When it happens

Trigger: Calling rewriteDataFiles(table).execute() while another job concurrently appends/deletes/compacts the same table, so the rewrite's validation/commit fails because the snapshot changed underneath it.

Common situations: Concurrent ingestion jobs (Flink/Spark streaming) running during a manual compaction; expired-snapshot cleanup running in parallel; long-running rewrite on a busy table committing against a moved snapshot.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/2031e1ceeb260e3f. Report an issue: GitHub.