apache/iceberg · error · RuntimeException

Cannot commit rewrite because of a ValidationException or Co

Error message

Cannot commit rewrite because of a ValidationException or CommitFailedException. This usually means that this rewrite has conflicted with another concurrent Iceberg operation. To reduce the likelihood of conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s. Separate smaller rewrite commits can succeed independently while any commits that conflict with another Iceberg operation will be ignored. This mode will create additional snapshots in the table history, one for each commit.

What it means

RewriteDataFiles doExecute rethrows ValidationException/CommitFailedException as RuntimeException with guidance: a rewrite group commit conflicted with concurrent table changes. The action suggests enabling partial progress so independent groups commit separately and conflicts are skipped.

Source

Thrown at spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java:315

          .run(commitManager::abortFileGroup);
      throw e;
    } finally {
      rewriteService.shutdown();
    }

    try {
      commitManager.commitOrClean(Sets.newHashSet(rewrittenGroups));
    } catch (ValidationException | CommitFailedException e) {
      String errorMessage =
          String.format(
              "Cannot commit rewrite because of a ValidationException or CommitFailedException. This usually means that "
                  + "this rewrite has conflicted with another concurrent Iceberg operation. To reduce the likelihood of "
                  + "conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s. "
                  + "Separate smaller rewrite commits can succeed independently while any commits that conflict with "
                  + "another Iceberg operation will be ignored. This mode will create additional snapshots in the table "
                  + "history, one for each commit.",
              PARTIAL_PROGRESS_ENABLED, PARTIAL_PROGRESS_MAX_COMMITS);
      throw new RuntimeException(errorMessage, e);
    }

    List<FileGroupRewriteResult> rewriteResults =
        rewrittenGroups.stream().map(RewriteFileGroup::asResult).collect(Collectors.toList());
    return ImmutableRewriteDataFiles.Result.builder().rewriteResults(rewriteResults);
  }

  private Builder doExecuteWithPartialProgress(
      FileRewritePlan<FileGroupInfo, FileScanTask, DataFile, RewriteFileGroup> plan,
      RewriteDataFilesCommitManager commitManager) {
    ExecutorService rewriteService = rewriteService();

    // start commit service
    int groupsPerCommit = IntMath.divide(plan.totalGroupCount(), maxCommits, RoundingMode.CEILING);
    RewriteDataFilesCommitManager.CommitService commitService =
        commitManager.service(groupsPerCommit);
    commitService.start();

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Enable partial progress: set rewrite.partial-progress.enabled=true (action: .option(PARTIAL_PROGRESS_ENABLED, "true")) so groups commit independently
2. Tune rewrite.partial-progress.max-commits to split the rewrite into more, smaller commits
3. Re-run the rewrite; conflicts are transient and re-running usually succeeds
4. Reduce concurrency on the table or coordinate compaction scheduling with writers

Example fix

// before (default: single commit, one conflict fails all)
SparkActions.get(spark).rewriteDataFiles(table).execute();
// after
SparkActions.get(spark).rewriteDataFiles(table)
    .option("rewrite.partial-progress.enabled", "true")
    .option("rewrite.partial-progress.max-commits", "10")
    .execute();

boolean concurrentWriters = checkRecentSnapshots(table);
if (concurrentWriters) { enablePartialProgressOptions(); }

try { action.execute(); } catch (RuntimeException e) { /* check cause is CommitFailed/Validation, re-run with partial progress enabled */ }

When it happens

Trigger: Expire/append/overwrite/compaction by another job advanced the table snapshot while this rewrite group was committing, causing CommitFailedException (or a validation failure) retried up to the max retries without success.

Common situations: Concurrent writes/compaction on the same table; long-running rewrite on a busy streaming table; single-commit (default partial progress disabled) mode where one conflict aborts everything.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Cannot commit rewrite because of a ValidationException or Co
• Cannot commit rewrite because of a ValidationException or Co
• Cannot commit rewrite because of a ValidationException or Co
• Cannot commit rewrite because of a ValidationException or Co
• Cannot commit rewrite because of a ValidationException or Co

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/689fb991b84fad6c. Report an issue: GitHub.