{"record":{"id":"b27eb26aaa4818ea","repo":"apache/iceberg","slug":"cannot-commit-rewrite-because-of-a-validationexcep-b27eb2","errorCode":null,"errorMessage":"Cannot commit rewrite because of a ValidationException or CommitFailedException. This usually means that this rewrite has conflicted with another concurrent Iceberg operation. To reduce the likelihood of conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s. Separate smaller rewrite commits can succeed independently while any commits that conflict with another Iceberg operation will be ignored. This mode will create additional snapshots in the table history, one for each commit.","messagePattern":"Cannot commit rewrite because of a ValidationException or CommitFailedException\\. This usually means that this rewrite has conflicted with another concurrent Iceberg operation\\. To reduce the likelihood of conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s\\. Separate smaller rewrite commits can succeed independently while any commits that conflict with another Iceberg operation will be ignored\\. This mode will create additional snapshots in the table history, one for each commit\\.","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/RewritePositionDeleteFilesSparkAction.java","lineNumber":248,"sourceCode":"      Tasks.foreach(rewrittenGroups).suppressFailureWhenFinished().run(commitManager::abort);\n      throw e;\n    } finally {\n      rewriteService.shutdown();\n    }\n\n    try {\n      commitManager.commitOrClean(Sets.newHashSet(rewrittenGroups));\n    } catch (ValidationException | CommitFailedException e) {\n      String errorMessage =\n          String.format(\n              \"Cannot commit rewrite because of a ValidationException or CommitFailedException. This usually means that \"\n                  + \"this rewrite has conflicted with another concurrent Iceberg operation. To reduce the likelihood of \"\n                  + \"conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s. \"\n                  + \"Separate smaller rewrite commits can succeed independently while any commits that conflict with \"\n                  + \"another Iceberg operation will be ignored. This mode will create additional snapshots in the table \"\n                  + \"history, one for each commit.\",\n              PARTIAL_PROGRESS_ENABLED, PARTIAL_PROGRESS_MAX_COMMITS);\n      throw new RuntimeException(errorMessage, e);\n    }\n\n    List<FileGroupRewriteResult> rewriteResults =\n        rewrittenGroups.stream()\n            .map(RewritePositionDeletesGroup::asResult)\n            .collect(Collectors.toList());\n\n    return ImmutableRewritePositionDeleteFiles.Result.builder()\n        .rewriteResults(rewriteResults)\n        .build();\n  }\n\n  private Result doExecuteWithPartialProgress(\n      FileRewritePlan<\n              FileGroupInfo, PositionDeletesScanTask, DeleteFile, RewritePositionDeletesGroup>\n          plan,\n      RewritePositionDeletesCommitManager commitManager) {\n    ExecutorService rewriteService = rewriteService();","sourceCodeStart":230,"sourceCodeEnd":266,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/RewritePositionDeleteFilesSparkAction.java#L230-L266","documentation":"RewritePositionDeleteFilesSparkAction wraps ValidationException or CommitFailedException in a RuntimeException when the rewrite commit fails. The message explains the rewrite conflicted with a concurrent Iceberg operation and suggests enabling partial progress (PARTIAL_PROGRESS_ENABLED with PARTIAL_PROGRESS_MAX_COMMITS) so the rewrite commits in smaller independent chunks.","triggerScenarios":"Executing rewritePositionDeletes() while another job concurrently commits to the same table (expires snapshots, compaction, appends), causing the final commit to fail validation or conflict and retry exhaustion.","commonSituations":"Two maintenance jobs (rewrite + expireSnapshots or rewriteDataFiles) scheduled at the same time; streaming writers committing while a batch rewrite finishes; long-running rewrites on busy tables.","solutions":["Enable partial progress: .option(PARTIAL_PROGRESS_ENABLED, 'true') and set PARTIAL_PROGRESS_MAX_COMMITS","Re-run the rewrite — commit conflicts are often transient; the next attempt sees the newer table state","Schedule the rewrite to avoid overlap with other table-maintenance or write jobs","Reduce rewrite duration (smaller groups / more parallelism) to shrink the conflict window"],"exampleFix":"// before\nSparkActions.get(table).rewritePositionDeletes().execute();\n// after\nSparkActions.get(table).rewritePositionDeletes()\n    .option(RewritePositionDeleteFilesSparkAction.PARTIAL_PROGRESS_ENABLED, \"true\")\n    .option(RewritePositionDeleteFilesSparkAction.PARTIAL_PROGRESS_MAX_COMMITS, \"10\")\n    .execute();","handlingStrategy":"retry","validationCode":"// check for concurrent commits before rewriting\nlong before = table.currentSnapshot().snapshotId();\n// ensure no other maintenance job holds a lock / is scheduled now","typeGuard":null,"tryCatchPattern":"try {\n  actions.rewritePositionDeletes().execute();\n} catch (RuntimeException e) {\n  if (e.getCause() instanceof ValidationException || e.getCause() instanceof CommitFailedException) {\n    // retry later or enable partial progress\n  } else { throw e; }\n}","preventionTips":["Enable PARTIAL_PROGRESS_ENABLED so conflicting commits are skipped instead of failing the whole rewrite","Schedule rewrites outside windows used by other maintenance/streaming commits","Retry failed rewrites — conflicts are usually transient","Keep rewrite runs short to reduce the conflict window"],"tags":["spark","commit-conflict","concurrency","position-deletes","rewrite"],"backgroundTag":"invalid-state-transition","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}