apache/iceberg · error · RuntimeException

Cannot commit rewrite because of a ValidationException or Co

Error message

Cannot commit rewrite because of a ValidationException or CommitFailedException. This usually means that this rewrite has conflicted with another concurrent Iceberg operation. To reduce the likelihood of conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s. Separate smaller rewrite commits can succeed independently while any commits that conflict with another Iceberg operation will be ignored. This mode will create additional snapshots in the table history, one for each commit.

What it means

RewritePositionDeleteFiles commits all rewrite groups in one transaction unless partial progress is enabled. A ValidationException or CommitFailedException escaping the commit means the rewrite conflicted with another concurrent table operation; the action wraps it in a RuntimeException with guidance to enable partial-progress commits.

Source

Thrown at spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/RewritePositionDeleteFilesSparkAction.java:248

      Tasks.foreach(rewrittenGroups).suppressFailureWhenFinished().run(commitManager::abort);
      throw e;
    } finally {
      rewriteService.shutdown();
    }

    try {
      commitManager.commitOrClean(Sets.newHashSet(rewrittenGroups));
    } catch (ValidationException | CommitFailedException e) {
      String errorMessage =
          String.format(
              "Cannot commit rewrite because of a ValidationException or CommitFailedException. This usually means that "
                  + "this rewrite has conflicted with another concurrent Iceberg operation. To reduce the likelihood of "
                  + "conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s. "
                  + "Separate smaller rewrite commits can succeed independently while any commits that conflict with "
                  + "another Iceberg operation will be ignored. This mode will create additional snapshots in the table "
                  + "history, one for each commit.",
              PARTIAL_PROGRESS_ENABLED, PARTIAL_PROGRESS_MAX_COMMITS);
      throw new RuntimeException(errorMessage, e);
    }

    List<FileGroupRewriteResult> rewriteResults =
        rewrittenGroups.stream()
            .map(RewritePositionDeletesGroup::asResult)
            .collect(Collectors.toList());

    return ImmutableRewritePositionDeleteFiles.Result.builder()
        .rewriteResults(rewriteResults)
        .build();
  }

  private Result doExecuteWithPartialProgress(
      FileRewritePlan<
              FileGroupInfo, PositionDeletesScanTask, DeleteFile, RewritePositionDeletesGroup>
          plan,
      RewritePositionDeletesCommitManager commitManager) {
    ExecutorService rewriteService = rewriteService();

View on GitHub (pinned to 86d9c8fc54)

Solutions

  1. Enable partial-progress.enabled so rewrites are committed in smaller independent groups
  2. Tune partial-progress.max-commits to control how the rewrite is split into commits
  3. Serialize conflicting jobs — avoid running delete-file rewrite concurrently with other rewrites/expiry, then re-run

Example fix

// before
actions.rewritePositionDeletes(table).execute();
// after
actions.rewritePositionDeletes(table)
    .option("partial-progress.enabled", "true")
    .option("partial-progress.max-commits", "10")
    .execute();
Defensive patterns

Strategy: try-catch

Validate before calling

// ensure no concurrent table mutations
boolean contended = concurrentJobsActive(table);
if (contended) { enablePartialProgress(); }

Try / catch

try { action.execute(); } catch (RuntimeException e) { if (e.getCause() instanceof ValidationException || e.getCause() instanceof CommitFailedException) { /* retry with partial-progress.enabled */ } }

Prevention

When it happens

Trigger: doExecute (invoked via execute) commits rewritten position-delete groups while another concurrent Iceberg operation changed the table, producing a ValidationException (requirement conflict) or exhausted CommitFailedException retries.

Common situations: Concurrent compaction/writes to the same table while deleting rewrite is running; long-running rewrite whose planned requirements were invalidated; repeated commit retries failing under contention.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/d5028a0ad5b00c41. Report an issue: GitHub.