{"record":{"id":"80d46607f007c907","repo":"apache/iceberg","slug":"is-true-but-rewrite-commits-failed-check-th-80d466","errorCode":null,"errorMessage":"{} is true but {} rewrite commits failed. Check the logs to determine why the individual commits failed. If this is persistent it may help to increase {} which will split the rewrite operation into smaller commits.","messagePattern":"(.+?) is true but (.+?) rewrite commits failed\\. Check the logs to determine why the individual commits failed\\. If this is persistent it may help to increase (.+?) which will split the rewrite operation into smaller commits\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java","lineNumber":358,"sourceCode":"        .onFailure(\n            (fileGroup, exception) -> {\n              LOG.error(\"Failure during rewrite group {}\", fileGroup.info(), exception);\n              rewriteFailures.add(\n                  ImmutableRewriteDataFiles.FileGroupFailureResult.builder()\n                      .info(fileGroup.info())\n                      .dataFilesCount(fileGroup.inputFileNum())\n                      .build());\n            })\n        .run(fileGroup -> commitService.offer(rewriteFiles(plan, fileGroup)));\n    rewriteService.shutdown();\n\n    // stop commit service\n    commitService.close();\n\n    int totalCommits = Math.min(plan.totalGroupCount(), maxCommits);\n    int failedCommits = totalCommits - commitService.succeededCommits();\n    if (failedCommits > 0 && failedCommits <= maxFailedCommits) {\n      LOG.warn(\n          \"{} is true but {} rewrite commits failed. Check the logs to determine why the individual \"\n              + \"commits failed. If this is persistent it may help to increase {} which will split the rewrite operation \"\n              + \"into smaller commits.\",\n          PARTIAL_PROGRESS_ENABLED,\n          failedCommits,\n          PARTIAL_PROGRESS_MAX_COMMITS);\n    } else if (failedCommits > maxFailedCommits) {\n      String errorMessage =\n          String.format(\n              Locale.ROOT,\n              \"%s is true but %d rewrite commits failed. This is more than the maximum allowed failures of %d. \"\n                  + \"Check the logs to determine why the individual commits failed. If this is persistent it may help to \"\n                  + \"increase %s which will split the rewrite operation into smaller commits.\",\n              PARTIAL_PROGRESS_ENABLED,\n              failedCommits,\n              maxFailedCommits,\n              PARTIAL_PROGRESS_MAX_COMMITS);\n      throw new RuntimeException(errorMessage);","sourceCodeStart":340,"sourceCodeEnd":376,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java#L340-L376","documentation":"WARN log emitted after a partial-progress rewrite finishes: partial-progress.enabled=true allows some commits to fail up to partial-progress.max-failed-commits, and the action reports that N of the total commits failed but the rewrite still completed partially. It is informational about degraded (not failed) results.","triggerScenarios":"RewriteDataFilesSparkAction.doExecuteWithPartialProgress where commitService.succeededCommits() is less than totalCommits (min of plan.totalGroupCount() and maxCommits) and failedCommits <= maxFailedCommits — concurrent table updates invalidated some rewrite commits.","commonSituations":"Concurrent writers appending/overwriting snapshots while compaction commits, causing ValidateMetrics/commit conflicts; snapshot expiry removing needed snapshots; optimistic concurrency clashes under heavy load.","solutions":["Check earlier WARN/error logs for each failed commit's reason.","Increase partial-progress.max-commits so each commit is smaller and less likely to conflict.","Re-run the rewrite to compact files missed by the failed commits.","Reduce concurrent write load or serialize compaction with other writers."],"exampleFix":"// before\naction.option(\"partial-progress.enabled\", \"true\").execute();\n// after: smaller commits reduce conflicts\naction.option(\"partial-progress.enabled\", \"true\")\n      .option(\"partial-progress.max-commits\", \"50\")\n      .execute();","handlingStrategy":"retry","validationCode":"boolean partial = Boolean.parseBoolean(table.properties().getOrDefault(\"write.spark.partial-progress.enabled\", \"false\")); int maxCommits = Integer.parseInt(table.properties().getOrDefault(\"write.spark.partial-progress.max-commits\", \"10\"));","typeGuard":null,"tryCatchPattern":"try { result = action.execute(); if (result.failedDataFilesCount() > 0) { /* log and schedule re-run */ } } catch (Exception e) { ... }","preventionTips":["Raise partial-progress.max-commits under concurrent-writer workloads","Schedule compaction outside peak concurrent-write windows","Re-run rewrite after partial failures to finish missed groups"],"tags":["logging","spark","commit-conflict","partial-progress"],"backgroundTag":"concurrent-commit-conflict","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}