{"record":{"id":"87e5fe530fe01489","repo":"apache/iceberg","slug":"cannot-commit-rewrite-because-of-a-validationexcep-87e5fe","errorCode":null,"errorMessage":"Cannot commit rewrite because of a ValidationException or CommitFailedException. This usually means that this rewrite has conflicted with another concurrent Iceberg operation. To reduce the likelihood of conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s. Separate smaller rewrite commits can succeed independently while any commits that conflict with another Iceberg operation will be ignored. This mode will create additional snapshots in the table history, one for each commit.","messagePattern":"Cannot commit rewrite because of a ValidationException or CommitFailedException\\. This usually means that this rewrite has conflicted with another concurrent Iceberg operation\\. To reduce the likelihood of conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s\\. Separate smaller rewrite commits can succeed independently while any commits that conflict with another Iceberg operation will be ignored\\. This mode will create additional snapshots in the table history, one for each commit\\.","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewritePositionDeleteFilesSparkAction.java","lineNumber":248,"sourceCode":"      Tasks.foreach(rewrittenGroups).suppressFailureWhenFinished().run(commitManager::abort);\n      throw e;\n    } finally {\n      rewriteService.shutdown();\n    }\n\n    try {\n      commitManager.commitOrClean(Sets.newHashSet(rewrittenGroups));\n    } catch (ValidationException | CommitFailedException e) {\n      String errorMessage =\n          String.format(\n              \"Cannot commit rewrite because of a ValidationException or CommitFailedException. This usually means that \"\n                  + \"this rewrite has conflicted with another concurrent Iceberg operation. To reduce the likelihood of \"\n                  + \"conflicts, set %s which will break up the rewrite into multiple smaller commits controlled by %s. \"\n                  + \"Separate smaller rewrite commits can succeed independently while any commits that conflict with \"\n                  + \"another Iceberg operation will be ignored. This mode will create additional snapshots in the table \"\n                  + \"history, one for each commit.\",\n              PARTIAL_PROGRESS_ENABLED, PARTIAL_PROGRESS_MAX_COMMITS);\n      throw new RuntimeException(errorMessage, e);\n    }\n\n    List<FileGroupRewriteResult> rewriteResults =\n        rewrittenGroups.stream()\n            .map(RewritePositionDeletesGroup::asResult)\n            .collect(Collectors.toList());\n\n    return ImmutableRewritePositionDeleteFiles.Result.builder()\n        .rewriteResults(rewriteResults)\n        .build();\n  }\n\n  private Result doExecuteWithPartialProgress(\n      FileRewritePlan<\n              FileGroupInfo, PositionDeletesScanTask, DeleteFile, RewritePositionDeletesGroup>\n          plan,\n      RewritePositionDeletesCommitManager commitManager) {\n    ExecutorService rewriteService = rewriteService();","sourceCodeStart":230,"sourceCodeEnd":266,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewritePositionDeleteFilesSparkAction.java#L230-L266","documentation":"In RewritePositionDeleteFilesSparkAction.doExecute, rewriting position-delete files failed with ValidationException or CommitFailedException. Like the data-file rewrite, the action wraps it in a RuntimeException explaining the rewrite conflicted with a concurrent Iceberg operation and pointing to partial-progress options.","triggerScenarios":"Calling rewritePositionDeleteFiles(table).execute() while concurrent writers add/remove delete files or commit snapshots, so group commits fail validation or conflict at commit time.","commonSituations":"Dangling-delete cleanup running alongside Flink upsert writers; concurrent compaction of delete files from two jobs; long rewrites on tables with frequent snapshot production.","solutions":["Enable partial progress: .option(\"partial-progress.enabled\", \"true\") and tune partial-progress.max-commits.","Run the rewrite when delete-file writers are quiesced, or retry during a quiet window.","Re-run after failure — already-committed groups are preserved; only failed groups need repeating.","Check the chained ValidationException if the conflict is a deterministic validation (e.g. sequence-number requirement) rather than a race."],"exampleFix":"// before\nSparkActions.get(spark).rewritePositionDeleteFiles(table).execute();\n// after\nSparkActions.get(spark).rewritePositionDeleteFiles(table)\n    .option(\"partial-progress.enabled\", \"true\")\n    .option(\"partial-progress.max-commits\", \"10\")\n    .execute();","handlingStrategy":"retry","validationCode":"// Check delete-file churn before rewrite\nlong deleteFiles = table.currentSnapshot().deleteManifests(table.io()).stream()\n    .mapToLong(m -> m.addedFilesCount() != null ? m.addedFilesCount() : 0).sum();\nLOG.info(\"{} delete files present; schedule rewrite during quiet window if churn is high\", deleteFiles);","typeGuard":null,"tryCatchPattern":"try {\n  rewrite.execute();\n} catch (RuntimeException e) {\n  if (e.getCause() instanceof ValidationException || e.getCause() instanceof CommitFailedException) {\n    // re-run with partial-progress.enabled=true or during quiet window\n  } else throw e;\n}","preventionTips":["Enable partial-progress mode for delete-file rewrites on streaming/upsert tables.","Coordinate with Flink/Spark writers to avoid rewriting deletes while they commit.","Retry failed rewrites — completed groups persist and only conflicts repeat."],"tags":["spark","position-deletes","commit-conflict","concurrency"],"backgroundTag":"commit-conflict","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}