{"record":{"id":"7958cf6a18958f11","repo":"apache/iceberg","slug":"is-true-but-rewrite-commits-failed-check-th-7958cf","errorCode":null,"errorMessage":"{} is true but {} rewrite commits failed. Check the logs to determine why the individual commits failed. If this is persistent it may help to increase {} which will split the rewrite operation into smaller commits.","messagePattern":"(.+?) is true but (.+?) rewrite commits failed\\. Check the logs to determine why the individual commits failed\\. If this is persistent it may help to increase (.+?) which will split the rewrite operation into smaller commits\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java","lineNumber":365,"sourceCode":"        .onFailure(\n            (fileGroup, exception) -> {\n              LOG.error(\"Failure during rewrite group {}\", fileGroup.info(), exception);\n              rewriteFailures.add(\n                  ImmutableRewriteDataFiles.FileGroupFailureResult.builder()\n                      .info(fileGroup.info())\n                      .dataFilesCount(fileGroup.inputFileNum())\n                      .build());\n            })\n        .run(fileGroup -> commitService.offer(rewriteFiles(plan, fileGroup)));\n    rewriteService.shutdown();\n\n    // stop commit service\n    commitService.close();\n\n    int totalCommits = Math.min(plan.totalGroupCount(), maxCommits);\n    int failedCommits = totalCommits - commitService.succeededCommits();\n    if (failedCommits > 0 && failedCommits <= maxFailedCommits) {\n      LOG.warn(\n          \"{} is true but {} rewrite commits failed. Check the logs to determine why the individual \"\n              + \"commits failed. If this is persistent it may help to increase {} which will split the rewrite operation \"\n              + \"into smaller commits.\",\n          PARTIAL_PROGRESS_ENABLED,\n          failedCommits,\n          PARTIAL_PROGRESS_MAX_COMMITS);\n    } else if (failedCommits > maxFailedCommits) {\n      String errorMessage =\n          String.format(\n              Locale.ROOT,\n              \"%s is true but %d rewrite commits failed. This is more than the maximum allowed failures of %d. \"\n                  + \"Check the logs to determine why the individual commits failed. If this is persistent it may help to \"\n                  + \"increase %s which will split the rewrite operation into smaller commits.\",\n              PARTIAL_PROGRESS_ENABLED,\n              failedCommits,\n              maxFailedCommits,\n              PARTIAL_PROGRESS_MAX_COMMITS);\n      throw new RuntimeException(errorMessage);","sourceCodeStart":347,"sourceCodeEnd":383,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java#L347-L383","documentation":"RewriteDataFilesSparkAction with partial progress enabled logs this warning when some rewrite group commits failed but stayed within max-failed-commits, so the action completes with partial results instead of failing. The library logs it because individual group commits can fail (e.g. concurrent table updates) while the overall operation still makes progress.","triggerScenarios":"Calling RewriteDataFilesSparkAction.execute() with partial-progress.enabled=true where totalCommits - commitService.succeededCommits() > 0 and <= partial-progress.max-commits in doExecuteWithPartialProgress.","commonSituations":"Concurrent writers commit to the table between rewrite planning and commit causing ValidationException; commit contention with compaction jobs; expired snapshots during long rewrites.","solutions":["Inspect executor/driver logs for the per-group commit exceptions","Increase partial-progress.max-commits to split work into smaller commits","Reduce concurrency with other writers or retry the rewrite after the conflicting commits finish","If failures are persistent and exceeded max-failed-commits, expect the action to fail; fix the underlying commit conflict first"],"exampleFix":"// before\nSparkActions.get(spark).rewriteDataFiles(table).option(\"partial-progress.enabled\", \"true\").execute();\n// after\nSparkActions.get(spark).rewriteDataFiles(table)\n    .option(\"partial-progress.enabled\", \"true\")\n    .option(\"partial-progress.max-commits\", \"20\")\n    .execute();","handlingStrategy":"validation","validationCode":"boolean partial = Boolean.parseBoolean(table.properties().getOrDefault(\"write.spark.partial-progress.enabled\", \"false\"));\nint maxCommits = Integer.parseInt(table.properties().getOrDefault(\"write.spark.partial-progress.max-commits\", \"10\"));\nif (partial && hasConcurrentWriters(table)) { /* schedule rewrite in a window with no other writers */ }","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Avoid running rewrites concurrently with streaming/other batch writers","Set partial-progress.max-commits proportional to table size","Monitor succeeded vs failed commit counts from action results","Check table snapshot history after rewrites to confirm expected commits"],"tags":["spark","compaction","commit-conflict","partial-progress"],"backgroundTag":"commit-conflict","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}