apache/iceberg · error · RuntimeException

Cannot commit rewrite because of a ValidationException or…

Error message

Cannot commit rewrite because of a ValidationException or CommitFailedException. This usually means that this rewrite has conflicted with another concurrent Iceberg operation. To reduce the likelihood of conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s. Separate smaller rewrite commits can succeed independently while any commits that conflict with another Iceberg operation will be ignored. This mode will create additional snapshots in the table history, one for each commit.

What it means

RewritePositionDeleteFilesSparkAction.doExecute wraps a ValidationException or CommitFailedException from the final table commit in a RuntimeException. The message explains that the rewrite conflicted with a concurrent Iceberg operation and suggests enabling partial progress mode (PARTIAL_PROGRESS_ENABLED / PARTIAL_PROGRESS_MAX_COMMITS) so smaller commits can succeed independently.

Solutions

  1. Enable partial progress: set rewrite.position-delete.partial-progress.enabled=true and rewrite.partial-progress.max-commits to a positive value so groups commit independently.
  2. Re-run the rewrite action; conflicting groups will be retried against the new table state.
  3. Schedule the rewrite to avoid concurrent write/maintenance jobs on the same table.

Example fix

// before
sparkActions.rewritePositionDeletes(table).execute();
// after
sparkActions.rewritePositionDeletes(table)
    .option("partial-progress.enabled", "true")
    .option("partial-progress.max-commits", "10")
    .execute();
Defensive patterns

Strategy: validation

Validate before calling

if (!table.properties().containsKey(RewritePositionDeleteFilesSparkAction.PARTIAL_PROGRESS_ENABLED)) { /* consider enabling partial progress on busy tables */ }

Try / catch

try { action.execute(); } catch (RuntimeException e) { if (e.getMessage() != null && e.getMessage().contains("Cannot commit rewrite")) { /* retry or enable partial progress */ } else throw e; }

Prevention

When it happens

Trigger: Running rewritePositionDeletes() while another concurrent operation (compaction, expireSnapshots, writes, another rewrite) commits to the same table, causing the single final commit to fail validation or commit-retry exhaustion.

Common situations: Long-running rewrites on actively written tables; parallel maintenance jobs; timeouts when many rewrites compete for commits.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/365573063665af33. Report an issue: GitHub.

Appendix: source

Thrown at spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/RewritePositionDeleteFilesSparkAction.java:248

      Tasks.foreach(rewrittenGroups).suppressFailureWhenFinished().run(commitManager::abort);
      throw e;
    } finally {
      rewriteService.shutdown();
    }

    try {
      commitManager.commitOrClean(Sets.newHashSet(rewrittenGroups));
    } catch (ValidationException | CommitFailedException e) {
      String errorMessage =
          String.format(
              "Cannot commit rewrite because of a ValidationException or CommitFailedException. This usually means that "
                  + "this rewrite has conflicted with another concurrent Iceberg operation. To reduce the likelihood of "
                  + "conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s. "
                  + "Separate smaller rewrite commits can succeed independently while any commits that conflict with "
                  + "another Iceberg operation will be ignored. This mode will create additional snapshots in the table "
                  + "history, one for each commit.",
              PARTIAL_PROGRESS_ENABLED, PARTIAL_PROGRESS_MAX_COMMITS);
      throw new RuntimeException(errorMessage, e);
    }

    List<FileGroupRewriteResult> rewriteResults =
        rewrittenGroups.stream()
            .map(RewritePositionDeletesGroup::asResult)
            .collect(Collectors.toList());

    return ImmutableRewritePositionDeleteFiles.Result.builder()
        .rewriteResults(rewriteResults)
        .build();
  }

  private Result doExecuteWithPartialProgress(
      FileRewritePlan<
              FileGroupInfo, PositionDeletesScanTask, DeleteFile, RewritePositionDeletesGroup>
          plan,
      RewritePositionDeletesCommitManager commitManager) {
    ExecutorService rewriteService = rewriteService();

View on GitHub (pinned to 86d9c8fc54)