{"record":{"id":"689fb991b84fad6c","repo":"apache/iceberg","slug":"cannot-commit-rewrite-because-of-a-validationexcep-689fb9","errorCode":null,"errorMessage":"Cannot commit rewrite because of a ValidationException or CommitFailedException. This usually means that this rewrite has conflicted with another concurrent Iceberg operation. To reduce the likelihood of conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s. Separate smaller rewrite commits can succeed independently while any commits that conflict with another Iceberg operation will be ignored. This mode will create additional snapshots in the table history, one for each commit.","messagePattern":"Cannot commit rewrite because of a ValidationException or CommitFailedException\\. This usually means that this rewrite has conflicted with another concurrent Iceberg operation\\. To reduce the likelihood of conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s\\. Separate smaller rewrite commits can succeed independently while any commits that conflict with another Iceberg operation will be ignored\\. This mode will create additional snapshots in the table history, one for each commit\\.","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java","lineNumber":315,"sourceCode":"          .run(commitManager::abortFileGroup);\n      throw e;\n    } finally {\n      rewriteService.shutdown();\n    }\n\n    try {\n      commitManager.commitOrClean(Sets.newHashSet(rewrittenGroups));\n    } catch (ValidationException | CommitFailedException e) {\n      String errorMessage =\n          String.format(\n              \"Cannot commit rewrite because of a ValidationException or CommitFailedException. This usually means that \"\n                  + \"this rewrite has conflicted with another concurrent Iceberg operation. To reduce the likelihood of \"\n                  + \"conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s. \"\n                  + \"Separate smaller rewrite commits can succeed independently while any commits that conflict with \"\n                  + \"another Iceberg operation will be ignored. This mode will create additional snapshots in the table \"\n                  + \"history, one for each commit.\",\n              PARTIAL_PROGRESS_ENABLED, PARTIAL_PROGRESS_MAX_COMMITS);\n      throw new RuntimeException(errorMessage, e);\n    }\n\n    List<FileGroupRewriteResult> rewriteResults =\n        rewrittenGroups.stream().map(RewriteFileGroup::asResult).collect(Collectors.toList());\n    return ImmutableRewriteDataFiles.Result.builder().rewriteResults(rewriteResults);\n  }\n\n  private Builder doExecuteWithPartialProgress(\n      FileRewritePlan<FileGroupInfo, FileScanTask, DataFile, RewriteFileGroup> plan,\n      RewriteDataFilesCommitManager commitManager) {\n    ExecutorService rewriteService = rewriteService();\n\n    // start commit service\n    int groupsPerCommit = IntMath.divide(plan.totalGroupCount(), maxCommits, RoundingMode.CEILING);\n    RewriteDataFilesCommitManager.CommitService commitService =\n        commitManager.service(groupsPerCommit);\n    commitService.start();\n","sourceCodeStart":297,"sourceCodeEnd":333,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java#L297-L333","documentation":"RewriteDataFiles doExecute rethrows ValidationException/CommitFailedException as RuntimeException with guidance: a rewrite group commit conflicted with concurrent table changes. The action suggests enabling partial progress so independent groups commit separately and conflicts are skipped.","triggerScenarios":"Expire/append/overwrite/compaction by another job advanced the table snapshot while this rewrite group was committing, causing CommitFailedException (or a validation failure) retried up to the max retries without success.","commonSituations":"Concurrent writes/compaction on the same table; long-running rewrite on a busy streaming table; single-commit (default partial progress disabled) mode where one conflict aborts everything.","solutions":["Enable partial progress: set rewrite.partial-progress.enabled=true (action: .option(PARTIAL_PROGRESS_ENABLED, \"true\")) so groups commit independently","Tune rewrite.partial-progress.max-commits to split the rewrite into more, smaller commits","Re-run the rewrite; conflicts are transient and re-running usually succeeds","Reduce concurrency on the table or coordinate compaction scheduling with writers"],"exampleFix":"// before (default: single commit, one conflict fails all)\nSparkActions.get(spark).rewriteDataFiles(table).execute();\n// after\nSparkActions.get(spark).rewriteDataFiles(table)\n    .option(\"rewrite.partial-progress.enabled\", \"true\")\n    .option(\"rewrite.partial-progress.max-commits\", \"10\")\n    .execute();","handlingStrategy":"retry","validationCode":"boolean concurrentWriters = checkRecentSnapshots(table);\nif (concurrentWriters) { enablePartialProgressOptions(); }","typeGuard":null,"tryCatchPattern":"try { action.execute(); } catch (RuntimeException e) { /* check cause is CommitFailed/Validation, re-run with partial progress enabled */ }","preventionTips":["Enable partial progress on busy tables","Schedule compaction away from heavy write windows","Keep rewrite groups small via max-commits","Monitor concurrent operations on the table"],"tags":["spark","rewrite-data-files","commit-conflict","concurrency"],"backgroundTag":"invalid-state-transition","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}