{"record":{"id":"b1d60dd5c12b310c","repo":"apache/iceberg","slug":"cannot-commit-rewrite-because-of-a-validationexcep-b1d60d","errorCode":null,"errorMessage":"Cannot commit rewrite because of a ValidationException or CommitFailedException. This usually means that this rewrite has conflicted with another concurrent Iceberg operation. To reduce the likelihood of conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s. Separate smaller rewrite commits can succeed independently while any commits that conflict with another Iceberg operation will be ignored. This mode will create additional snapshots in the table history, one for each commit.","messagePattern":"Cannot commit rewrite because of a ValidationException or CommitFailedException\\. This usually means that this rewrite has conflicted with another concurrent Iceberg operation\\. To reduce the likelihood of conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s\\. Separate smaller rewrite commits can succeed independently while any commits that conflict with another Iceberg operation will be ignored\\. This mode will create additional snapshots in the table history, one for each commit\\.","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java","lineNumber":322,"sourceCode":"          .run(commitManager::abortFileGroup);\n      throw e;\n    } finally {\n      rewriteService.shutdown();\n    }\n\n    try {\n      commitManager.commitOrClean(Sets.newHashSet(rewrittenGroups));\n    } catch (ValidationException | CommitFailedException e) {\n      String errorMessage =\n          String.format(\n              \"Cannot commit rewrite because of a ValidationException or CommitFailedException. This usually means that \"\n                  + \"this rewrite has conflicted with another concurrent Iceberg operation. To reduce the likelihood of \"\n                  + \"conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s. \"\n                  + \"Separate smaller rewrite commits can succeed independently while any commits that conflict with \"\n                  + \"another Iceberg operation will be ignored. This mode will create additional snapshots in the table \"\n                  + \"history, one for each commit.\",\n              PARTIAL_PROGRESS_ENABLED, PARTIAL_PROGRESS_MAX_COMMITS);\n      throw new RuntimeException(errorMessage, e);\n    }\n\n    List<FileGroupRewriteResult> rewriteResults =\n        rewrittenGroups.stream().map(RewriteFileGroup::asResult).collect(Collectors.toList());\n    return ImmutableRewriteDataFiles.Result.builder().rewriteResults(rewriteResults);\n  }\n\n  private Builder doExecuteWithPartialProgress(\n      FileRewritePlan<FileGroupInfo, FileScanTask, DataFile, RewriteFileGroup> plan,\n      RewriteDataFilesCommitManager commitManager) {\n    ExecutorService rewriteService = rewriteService();\n\n    // start commit service\n    int groupsPerCommit = IntMath.divide(plan.totalGroupCount(), maxCommits, RoundingMode.CEILING);\n    RewriteDataFilesCommitManager.CommitService commitService =\n        commitManager.service(groupsPerCommit);\n    commitService.start();\n","sourceCodeStart":304,"sourceCodeEnd":340,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java#L304-L340","documentation":"RewriteDataFiles commits all rewrite groups in a single transaction (unless partial progress is enabled). If any ValidationException or CommitFailedException escapes the commit, the action wraps it in a RuntimeException explaining the conflict and pointing to the partial-progress configuration as mitigation.","triggerScenarios":"doExecute commits rewritten file groups and a ValidationException (requirement conflict, e.g. current schema/snapshot changed) or a persistent CommitFailedException occurs because another concurrent operation modified the table.","commonSituations":"Concurrent writes/compactions/expireSnapshots running against the same table; retries exhausted due to sustained write contention; requirement validation failing because the table changed since planning.","solutions":["Enable partial progress: set rewrite.partials-progress.enabled (PARTIAL_PROGRESS_ENABLED) to true so commits happen in smaller independent groups","Tune PARTIAL_PROGRESS_MAX_COMMITS to split rewrites into multiple commits that can succeed independently","Reduce concurrent operations on the table or schedule compaction off-peak, then re-run the rewrite"],"exampleFix":"// before\nactions.rewriteDataFiles(table).execute();\n// after\nactions.rewriteDataFiles(table)\n    .option(\"partial-progress.enabled\", \"true\")\n    .option(\"partial-progress.max-commits\", \"10\")\n    .execute();","handlingStrategy":"try-catch","validationCode":"// avoid concurrent table mutations while rewrite runs\nboolean contended = concurrentJobsActive(table); // check via your job scheduler\nif (contended) { enablePartialProgress(); }","typeGuard":null,"tryCatchPattern":"try { action.execute(); } catch (RuntimeException e) { if (e.getCause() instanceof ValidationException || e.getCause() instanceof CommitFailedException) { /* re-run with partial-progress.enabled=true */ } }","preventionTips":["Enable partial-progress.enabled for contended tables","Schedule rewrites when concurrent writers are quiescent","Keep rewrite planning-to-commit window short"],"tags":["spark","iceberg","commit-conflict","concurrency"],"backgroundTag":"invalid-state-transition","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}