{"record":{"id":"446cfc7469a64845","repo":"apache/iceberg","slug":"cannot-commit-rewrite-because-of-a-validationexcep-446cfc","errorCode":null,"errorMessage":"Cannot commit rewrite because of a ValidationException or CommitFailedException. This usually means that this rewrite has conflicted with another concurrent Iceberg operation. To reduce the likelihood of conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s. Separate smaller rewrite commits can succeed independently while any commits that conflict with another Iceberg operation will be ignored. This mode will create additional snapshots in the table history, one for each commit.","messagePattern":"Cannot commit rewrite because of a ValidationException or CommitFailedException\\. This usually means that this rewrite has conflicted with another concurrent Iceberg operation\\. To reduce the likelihood of conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s\\. Separate smaller rewrite commits can succeed independently while any commits that conflict with another Iceberg operation will be ignored\\. This mode will create additional snapshots in the table history, one for each commit\\.","errorType":"exception","errorClass":"java.lang.RuntimeException","httpStatus":null,"severity":"error","filePath":"spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java","lineNumber":322,"sourceCode":"          .run(commitManager::abortFileGroup);\n      throw e;\n    } finally {\n      rewriteService.shutdown();\n    }\n\n    try {\n      commitManager.commitOrClean(Sets.newHashSet(rewrittenGroups));\n    } catch (ValidationException | CommitFailedException e) {\n      String errorMessage =\n          String.format(\n              \"Cannot commit rewrite because of a ValidationException or CommitFailedException. This usually means that \"\n                  + \"this rewrite has conflicted with another concurrent Iceberg operation. To reduce the likelihood of \"\n                  + \"conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s. \"\n                  + \"Separate smaller rewrite commits can succeed independently while any commits that conflict with \"\n                  + \"another Iceberg operation will be ignored. This mode will create additional snapshots in the table \"\n                  + \"history, one for each commit.\",\n              PARTIAL_PROGRESS_ENABLED, PARTIAL_PROGRESS_MAX_COMMITS);\n      throw new RuntimeException(errorMessage, e);\n    }\n\n    List<FileGroupRewriteResult> rewriteResults =\n        rewrittenGroups.stream().map(RewriteFileGroup::asResult).collect(Collectors.toList());\n    return ImmutableRewriteDataFiles.Result.builder().rewriteResults(rewriteResults);\n  }\n\n  private Builder doExecuteWithPartialProgress(\n      FileRewritePlan<FileGroupInfo, FileScanTask, DataFile, RewriteFileGroup> plan,\n      RewriteDataFilesCommitManager commitManager) {\n    ExecutorService rewriteService = rewriteService();\n\n    // start commit service\n    int groupsPerCommit = IntMath.divide(plan.totalGroupCount(), maxCommits, RoundingMode.CEILING);\n    RewriteDataFilesCommitManager.CommitService commitService =\n        commitManager.service(groupsPerCommit);\n    commitService.start();\n","sourceCodeStart":304,"sourceCodeEnd":340,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java#L304-L340","documentation":"RewriteDataFiles commits rewrite groups in a single commit when partial progress is disabled. If a ValidationException or CommitFailedException occurs (typically a concurrent Iceberg operation changed the table), the whole rewrite fails with this RuntimeException explaining that partial-progress mode would allow independent smaller commits.","triggerScenarios":"Running optimize/rewrite_data_files while concurrent compaction, expire_snapshots, or other commits modify the table, causing commit validation to fail; also equality-delete/sequence validation conflicts.","commonSituations":"Multiple jobs compacting the same table simultaneously; a stream (Flink/Spark) writing continuously while a rewrite commits; long-running rewrite colliding with scheduled maintenance.","solutions":["Enable partial progress: partial_progress.enabled=true (option partial_progress.enabled in SQL), which commits groups independently","Tune partial_progress.max_commits to control the number of commits","Schedule rewrites to avoid concurrent writers, or retry the rewrite after the conflicting operation completes"],"exampleFix":"// before\nCALL iceberg.system.rewrite_data_files(table => 'db.t')\n// after\nCALL iceberg.system.rewrite_data_files(\n  table => 'db.t',\n  options => map('partial-progress.enabled', 'true', 'partial-progress.max-commits', '10'));","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try { rewriteAction.execute(); } catch (RuntimeException e) { if (e.getMessage().contains(\"conflicted with another concurrent Iceberg operation\")) { /* re-run with partial-progress.enabled=true */ } }","preventionTips":["Enable partial-progress mode for rewrites on tables with concurrent writers","Schedule compaction windows that exclude other maintenance jobs","Watch for continuous streaming writers; keep rewrite commits small and frequent"],"tags":["spark","compaction","commit-conflict","concurrency"],"backgroundTag":"invalid-state-transition","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}