apache/iceberg · error · java.lang.RuntimeException

Cannot commit rewrite because of a ValidationException or Co

Error message

Cannot commit rewrite because of a ValidationException or CommitFailedException. This usually means that this rewrite has conflicted with another concurrent Iceberg operation. To reduce the likelihood of conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s. Separate smaller rewrite commits can succeed independently while any commits that conflict with another Iceberg operation will be ignored. This mode will create additional snapshots in the table history, one for each commit.

What it means

RewriteDataFiles commits rewrite groups in a single commit when partial progress is disabled. If a ValidationException or CommitFailedException occurs (typically a concurrent Iceberg operation changed the table), the whole rewrite fails with this RuntimeException explaining that partial-progress mode would allow independent smaller commits.

Source

Thrown at spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java:322

          .run(commitManager::abortFileGroup);
      throw e;
    } finally {
      rewriteService.shutdown();
    }

    try {
      commitManager.commitOrClean(Sets.newHashSet(rewrittenGroups));
    } catch (ValidationException | CommitFailedException e) {
      String errorMessage =
          String.format(
              "Cannot commit rewrite because of a ValidationException or CommitFailedException. This usually means that "
                  + "this rewrite has conflicted with another concurrent Iceberg operation. To reduce the likelihood of "
                  + "conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s. "
                  + "Separate smaller rewrite commits can succeed independently while any commits that conflict with "
                  + "another Iceberg operation will be ignored. This mode will create additional snapshots in the table "
                  + "history, one for each commit.",
              PARTIAL_PROGRESS_ENABLED, PARTIAL_PROGRESS_MAX_COMMITS);
      throw new RuntimeException(errorMessage, e);
    }

    List<FileGroupRewriteResult> rewriteResults =
        rewrittenGroups.stream().map(RewriteFileGroup::asResult).collect(Collectors.toList());
    return ImmutableRewriteDataFiles.Result.builder().rewriteResults(rewriteResults);
  }

  private Builder doExecuteWithPartialProgress(
      FileRewritePlan<FileGroupInfo, FileScanTask, DataFile, RewriteFileGroup> plan,
      RewriteDataFilesCommitManager commitManager) {
    ExecutorService rewriteService = rewriteService();

    // start commit service
    int groupsPerCommit = IntMath.divide(plan.totalGroupCount(), maxCommits, RoundingMode.CEILING);
    RewriteDataFilesCommitManager.CommitService commitService =
        commitManager.service(groupsPerCommit);
    commitService.start();

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Enable partial progress: partial_progress.enabled=true (option partial_progress.enabled in SQL), which commits groups independently
2. Tune partial_progress.max_commits to control the number of commits
3. Schedule rewrites to avoid concurrent writers, or retry the rewrite after the conflicting operation completes

Example fix

// before
CALL iceberg.system.rewrite_data_files(table => 'db.t')
// after
CALL iceberg.system.rewrite_data_files(
  table => 'db.t',
  options => map('partial-progress.enabled', 'true', 'partial-progress.max-commits', '10'));

try { rewriteAction.execute(); } catch (RuntimeException e) { if (e.getMessage().contains("conflicted with another concurrent Iceberg operation")) { /* re-run with partial-progress.enabled=true */ } }

When it happens

Trigger: Running optimize/rewrite_data_files while concurrent compaction, expire_snapshots, or other commits modify the table, causing commit validation to fail; also equality-delete/sequence validation conflicts.

Common situations: Multiple jobs compacting the same table simultaneously; a stream (Flink/Spark) writing continuously while a rewrite commits; long-running rewrite colliding with scheduled maintenance.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• %s is true but %d rewrite commits failed. This is more than
• Cannot commit rewrite because of a ValidationException or Co
• Cannot commit rewrite because of a ValidationException or Co
• Cannot commit rewrite because of a ValidationException or Co
• Cannot commit rewrite because of a ValidationException or Co

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/446cfc7469a64845. Report an issue: GitHub.