apache/iceberg · error · RuntimeException

Cannot commit rewrite because of a ValidationException or Co

Error message

Cannot commit rewrite because of a ValidationException or CommitFailedException. This usually means that this rewrite has conflicted with another concurrent Iceberg operation. To reduce the likelihood of conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s. Separate smaller rewrite commits can succeed independently while any commits that conflict with another Iceberg operation will be ignored. This mode will create additional snapshots in the table history, one for each commit.

What it means

RewriteDataFiles commits all rewrite groups in a single transaction (unless partial progress is enabled). If any ValidationException or CommitFailedException escapes the commit, the action wraps it in a RuntimeException explaining the conflict and pointing to the partial-progress configuration as mitigation.

Source

Thrown at spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java:322

          .run(commitManager::abortFileGroup);
      throw e;
    } finally {
      rewriteService.shutdown();
    }

    try {
      commitManager.commitOrClean(Sets.newHashSet(rewrittenGroups));
    } catch (ValidationException | CommitFailedException e) {
      String errorMessage =
          String.format(
              "Cannot commit rewrite because of a ValidationException or CommitFailedException. This usually means that "
                  + "this rewrite has conflicted with another concurrent Iceberg operation. To reduce the likelihood of "
                  + "conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s. "
                  + "Separate smaller rewrite commits can succeed independently while any commits that conflict with "
                  + "another Iceberg operation will be ignored. This mode will create additional snapshots in the table "
                  + "history, one for each commit.",
              PARTIAL_PROGRESS_ENABLED, PARTIAL_PROGRESS_MAX_COMMITS);
      throw new RuntimeException(errorMessage, e);
    }

    List<FileGroupRewriteResult> rewriteResults =
        rewrittenGroups.stream().map(RewriteFileGroup::asResult).collect(Collectors.toList());
    return ImmutableRewriteDataFiles.Result.builder().rewriteResults(rewriteResults);
  }

  private Builder doExecuteWithPartialProgress(
      FileRewritePlan<FileGroupInfo, FileScanTask, DataFile, RewriteFileGroup> plan,
      RewriteDataFilesCommitManager commitManager) {
    ExecutorService rewriteService = rewriteService();

    // start commit service
    int groupsPerCommit = IntMath.divide(plan.totalGroupCount(), maxCommits, RoundingMode.CEILING);
    RewriteDataFilesCommitManager.CommitService commitService =
        commitManager.service(groupsPerCommit);
    commitService.start();

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Enable partial progress: set rewrite.partials-progress.enabled (PARTIAL_PROGRESS_ENABLED) to true so commits happen in smaller independent groups
2. Tune PARTIAL_PROGRESS_MAX_COMMITS to split rewrites into multiple commits that can succeed independently
3. Reduce concurrent operations on the table or schedule compaction off-peak, then re-run the rewrite

Example fix

// before
actions.rewriteDataFiles(table).execute();
// after
actions.rewriteDataFiles(table)
    .option("partial-progress.enabled", "true")
    .option("partial-progress.max-commits", "10")
    .execute();

// avoid concurrent table mutations while rewrite runs
boolean contended = concurrentJobsActive(table); // check via your job scheduler
if (contended) { enablePartialProgress(); }

try { action.execute(); } catch (RuntimeException e) { if (e.getCause() instanceof ValidationException || e.getCause() instanceof CommitFailedException) { /* re-run with partial-progress.enabled=true */ } }

When it happens

Trigger: doExecute commits rewritten file groups and a ValidationException (requirement conflict, e.g. current schema/snapshot changed) or a persistent CommitFailedException occurs because another concurrent operation modified the table.

Common situations: Concurrent writes/compactions/expireSnapshots running against the same table; retries exhausted due to sustained write contention; requirement validation failing because the table changed since planning.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Cannot commit rewrite because of a ValidationException or Co
• Cannot commit rewrite because of a ValidationException or Co
• Cannot commit rewrite because of a ValidationException or Co
• Cannot commit rewrite because of a ValidationException or Co
• Cannot commit rewrite because of a ValidationException or Co

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/b1d60dd5c12b310c. Report an issue: GitHub.