apache/iceberg · error · RuntimeException

Cannot commit rewrite because of a ValidationException or Co

Error message

Cannot commit rewrite because of a ValidationException or CommitFailedException. This usually means that this rewrite has conflicted with another concurrent Iceberg operation. To reduce the likelihood of conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s. Separate smaller rewrite commits can succeed independently while any commits that conflict with another Iceberg operation will be ignored. This mode will create additional snapshots in the table history, one for each commit.

What it means

In RewritePositionDeleteFilesSparkAction.doExecute, rewriting position-delete files failed with ValidationException or CommitFailedException. Like the data-file rewrite, the action wraps it in a RuntimeException explaining the rewrite conflicted with a concurrent Iceberg operation and pointing to partial-progress options.

Source

Thrown at spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewritePositionDeleteFilesSparkAction.java:248

      Tasks.foreach(rewrittenGroups).suppressFailureWhenFinished().run(commitManager::abort);
      throw e;
    } finally {
      rewriteService.shutdown();
    }

    try {
      commitManager.commitOrClean(Sets.newHashSet(rewrittenGroups));
    } catch (ValidationException | CommitFailedException e) {
      String errorMessage =
          String.format(
              "Cannot commit rewrite because of a ValidationException or CommitFailedException. This usually means that "
                  + "this rewrite has conflicted with another concurrent Iceberg operation. To reduce the likelihood of "
                  + "conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s. "
                  + "Separate smaller rewrite commits can succeed independently while any commits that conflict with "
                  + "another Iceberg operation will be ignored. This mode will create additional snapshots in the table "
                  + "history, one for each commit.",
              PARTIAL_PROGRESS_ENABLED, PARTIAL_PROGRESS_MAX_COMMITS);
      throw new RuntimeException(errorMessage, e);
    }

    List<FileGroupRewriteResult> rewriteResults =
        rewrittenGroups.stream()
            .map(RewritePositionDeletesGroup::asResult)
            .collect(Collectors.toList());

    return ImmutableRewritePositionDeleteFiles.Result.builder()
        .rewriteResults(rewriteResults)
        .build();
  }

  private Result doExecuteWithPartialProgress(
      FileRewritePlan<
              FileGroupInfo, PositionDeletesScanTask, DeleteFile, RewritePositionDeletesGroup>
          plan,
      RewritePositionDeletesCommitManager commitManager) {
    ExecutorService rewriteService = rewriteService();

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Enable partial progress: .option("partial-progress.enabled", "true") and tune partial-progress.max-commits.
2. Run the rewrite when delete-file writers are quiesced, or retry during a quiet window.
3. Re-run after failure — already-committed groups are preserved; only failed groups need repeating.
4. Check the chained ValidationException if the conflict is a deterministic validation (e.g. sequence-number requirement) rather than a race.

Example fix

// before
SparkActions.get(spark).rewritePositionDeleteFiles(table).execute();
// after
SparkActions.get(spark).rewritePositionDeleteFiles(table)
    .option("partial-progress.enabled", "true")
    .option("partial-progress.max-commits", "10")
    .execute();

// Check delete-file churn before rewrite
long deleteFiles = table.currentSnapshot().deleteManifests(table.io()).stream()
    .mapToLong(m -> m.addedFilesCount() != null ? m.addedFilesCount() : 0).sum();
LOG.info("{} delete files present; schedule rewrite during quiet window if churn is high", deleteFiles);

try {
  rewrite.execute();
} catch (RuntimeException e) {
  if (e.getCause() instanceof ValidationException || e.getCause() instanceof CommitFailedException) {
    // re-run with partial-progress.enabled=true or during quiet window
  } else throw e;
}

When it happens

Trigger: Calling rewritePositionDeleteFiles(table).execute() while concurrent writers add/remove delete files or commit snapshots, so group commits fail validation or conflict at commit time.

Common situations: Dangling-delete cleanup running alongside Flink upsert writers; concurrent compaction of delete files from two jobs; long rewrites on tables with frequent snapshot production.

Related errors

• Cannot commit rewrite because of a ValidationException or Co
• Cannot commit rewrite because of a ValidationException or Co
• Cannot commit rewrite because of a ValidationException or Co
• Cannot commit rewrite because of a ValidationException or Co
• Cannot commit rewrite because of a ValidationException or Co

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/87e5fe530fe01489. Report an issue: GitHub.