apache/iceberg · error · RuntimeException

Cannot commit rewrite because of a ValidationException or Co

Error message

Cannot commit rewrite because of a ValidationException or CommitFailedException. This usually means that this rewrite has conflicted with another concurrent Iceberg operation. To reduce the likelihood of conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s. Separate smaller rewrite commits can succeed independently while any commits that conflict with another Iceberg operation will be ignored. This mode will create additional snapshots in the table history, one for each commit.

What it means

RewritePositionDeleteFilesSparkAction wraps ValidationException or CommitFailedException in a RuntimeException when the rewrite commit fails. The message explains the rewrite conflicted with a concurrent Iceberg operation and suggests enabling partial progress (PARTIAL_PROGRESS_ENABLED with PARTIAL_PROGRESS_MAX_COMMITS) so the rewrite commits in smaller independent chunks.

Source

Thrown at spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/RewritePositionDeleteFilesSparkAction.java:248

      Tasks.foreach(rewrittenGroups).suppressFailureWhenFinished().run(commitManager::abort);
      throw e;
    } finally {
      rewriteService.shutdown();
    }

    try {
      commitManager.commitOrClean(Sets.newHashSet(rewrittenGroups));
    } catch (ValidationException | CommitFailedException e) {
      String errorMessage =
          String.format(
              "Cannot commit rewrite because of a ValidationException or CommitFailedException. This usually means that "
                  + "this rewrite has conflicted with another concurrent Iceberg operation. To reduce the likelihood of "
                  + "conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s. "
                  + "Separate smaller rewrite commits can succeed independently while any commits that conflict with "
                  + "another Iceberg operation will be ignored. This mode will create additional snapshots in the table "
                  + "history, one for each commit.",
              PARTIAL_PROGRESS_ENABLED, PARTIAL_PROGRESS_MAX_COMMITS);
      throw new RuntimeException(errorMessage, e);
    }

    List<FileGroupRewriteResult> rewriteResults =
        rewrittenGroups.stream()
            .map(RewritePositionDeletesGroup::asResult)
            .collect(Collectors.toList());

    return ImmutableRewritePositionDeleteFiles.Result.builder()
        .rewriteResults(rewriteResults)
        .build();
  }

  private Result doExecuteWithPartialProgress(
      FileRewritePlan<
              FileGroupInfo, PositionDeletesScanTask, DeleteFile, RewritePositionDeletesGroup>
          plan,
      RewritePositionDeletesCommitManager commitManager) {
    ExecutorService rewriteService = rewriteService();

View on GitHub (pinned to 86d9c8fc54)

Solutions

Enable partial progress: .option(PARTIAL_PROGRESS_ENABLED, 'true') and set PARTIAL_PROGRESS_MAX_COMMITS
Re-run the rewrite — commit conflicts are often transient; the next attempt sees the newer table state
Schedule the rewrite to avoid overlap with other table-maintenance or write jobs
Reduce rewrite duration (smaller groups / more parallelism) to shrink the conflict window

Example fix

// before
SparkActions.get(table).rewritePositionDeletes().execute();
// after
SparkActions.get(table).rewritePositionDeletes()
    .option(RewritePositionDeleteFilesSparkAction.PARTIAL_PROGRESS_ENABLED, "true")
    .option(RewritePositionDeleteFilesSparkAction.PARTIAL_PROGRESS_MAX_COMMITS, "10")
    .execute();

Defensive patterns

Strategy: retry

Validate before calling

// check for concurrent commits before rewriting
long before = table.currentSnapshot().snapshotId();
// ensure no other maintenance job holds a lock / is scheduled now

Try / catch

try {
  actions.rewritePositionDeletes().execute();
} catch (RuntimeException e) {
  if (e.getCause() instanceof ValidationException || e.getCause() instanceof CommitFailedException) {
    // retry later or enable partial progress
  } else { throw e; }
}

Prevention

Enable PARTIAL_PROGRESS_ENABLED so conflicting commits are skipped instead of failing the whole rewrite
Schedule rewrites outside windows used by other maintenance/streaming commits
Retry failed rewrites — conflicts are usually transient
Keep rewrite runs short to reduce the conflict window

When it happens

Trigger: Executing rewritePositionDeletes() while another job concurrently commits to the same table (expires snapshots, compaction, appends), causing the final commit to fail validation or conflict and retry exhaustion.

Common situations: Two maintenance jobs (rewrite + expireSnapshots or rewriteDataFiles) scheduled at the same time; streaming writers committing while a batch rewrite finishes; long-running rewrites on busy tables.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/b27eb26aaa4818ea. Report an issue: GitHub.