{"record":{"id":"d4534be54512caa2","repo":"apache/iceberg","slug":"is-true-but-rewrite-commits-failed-check-th-d4534b","errorCode":null,"errorMessage":"{} is true but {} rewrite commits failed. Check the logs to determine why the individual commits failed. If this is persistent it may help to increase {} which will split the rewrite operation into smaller commits.","messagePattern":"(.+?) is true but (.+?) rewrite commits failed\\. Check the logs to determine why the individual commits failed\\. If this is persistent it may help to increase (.+?) which will split the rewrite operation into smaller commits\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java","lineNumber":365,"sourceCode":"        .onFailure(\n            (fileGroup, exception) -> {\n              LOG.error(\"Failure during rewrite group {}\", fileGroup.info(), exception);\n              rewriteFailures.add(\n                  ImmutableRewriteDataFiles.FileGroupFailureResult.builder()\n                      .info(fileGroup.info())\n                      .dataFilesCount(fileGroup.inputFileNum())\n                      .build());\n            })\n        .run(fileGroup -> commitService.offer(rewriteFiles(plan, fileGroup)));\n    rewriteService.shutdown();\n\n    // stop commit service\n    commitService.close();\n\n    int totalCommits = Math.min(plan.totalGroupCount(), maxCommits);\n    int failedCommits = totalCommits - commitService.succeededCommits();\n    if (failedCommits > 0 && failedCommits <= maxFailedCommits) {\n      LOG.warn(\n          \"{} is true but {} rewrite commits failed. Check the logs to determine why the individual \"\n              + \"commits failed. If this is persistent it may help to increase {} which will split the rewrite operation \"\n              + \"into smaller commits.\",\n          PARTIAL_PROGRESS_ENABLED,\n          failedCommits,\n          PARTIAL_PROGRESS_MAX_COMMITS);\n    } else if (failedCommits > maxFailedCommits) {\n      String errorMessage =\n          String.format(\n              Locale.ROOT,\n              \"%s is true but %d rewrite commits failed. This is more than the maximum allowed failures of %d. \"\n                  + \"Check the logs to determine why the individual commits failed. If this is persistent it may help to \"\n                  + \"increase %s which will split the rewrite operation into smaller commits.\",\n              PARTIAL_PROGRESS_ENABLED,\n              failedCommits,\n              maxFailedCommits,\n              PARTIAL_PROGRESS_MAX_COMMITS);\n      throw new RuntimeException(errorMessage);","sourceCodeStart":347,"sourceCodeEnd":383,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java#L347-L383","documentation":"A logged warning emitted by RewriteDataFilesSparkAction.doExecuteWithPartialProgress when partial-progress mode is enabled but some rewrite group commits failed while staying within the allowed maxFailedCommits. The overall action still succeeds with the groups that committed; this warns that compaction was incomplete and suggests raising partial-progress.max-commits so rewrites split into smaller commits.","triggerScenarios":"Running rewrite_data_files with partial_progress.enabled=true when concurrent commits (e.g., from other jobs) cause RewriteFileGroup commit failures — CommitStateUnknownException, validation failures against a moved base table snapshot, or catalog conflicts — but failedCommits <= partial-progress.max-commits.","commonSituations":"Concurrent writes/compaction on the same table causing commit conflicts; REST catalog rate-limiting or flakiness during commit bursts; too few max-commits so each commit carries too many groups and conflicts more often; long-running rewrites while the table's latest snapshot advances.","solutions":["Increase partial-progress.max-commits so each commit contains fewer groups and conflicts less.","Retry the rewrite; only groups whose commits failed remain un-compacted.","Reduce concurrent writers during compaction, or serialize maintenance jobs.","Check individual commit failure logs (validation failures, catalog errors) and address the root cause.","If conflicts persist, lower the group count or run rewrites during quiescent windows."],"exampleFix":"// before\nrewrite_data_files(partial_progress.enabled = true, partial_progress.max_commits = 1)\n// after\nrewrite_data_files(\n  partial_progress.enabled = true,\n  partial_progress.max_commits = 10  // smaller commits, fewer conflicts\n)","handlingStrategy":"retry","validationCode":"// check table-level commit contention before running partial-progress rewrites\nTable table = catalog.loadTable(TableIdentifier.of(\"db\", \"tbl\"));\nint recentSnapshots = java.util.Lists.newArrayList(table.snapshots()).size(); // ensure no heavy concurrent writers","typeGuard":null,"tryCatchPattern":"try { rewrite.execute(); } catch (Exception e) { /* commits within maxFailedCommits are tolerated; re-run to finish */ }","preventionTips":["Raise partial-progress.max-commits to shrink per-commit conflict surface","Schedule rewrites during quiescent windows without concurrent writers","Investigate individual commit failure logs first","Serialize maintenance jobs on the same table"],"tags":["spark","compaction","commit-conflict","partial-progress"],"backgroundTag":"invalid-state-transition","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}