{"record":{"id":"06fb268430edaf0e","repo":"apache/iceberg","slug":"s-is-true-but-d-rewrite-commits-failed-this-is-06fb26","errorCode":null,"errorMessage":"%s is true but %d rewrite commits failed. This is more than the maximum allowed failures of %d. Check the logs to determine why the individual commits failed. If this is persistent it may help to increase %s which will split the rewrite operation into smaller commits.","messagePattern":"(.+?) is true but (.+?) rewrite commits failed\\. This is more than the maximum allowed failures of (.+?)\\. Check the logs to determine why the individual commits failed\\. If this is persistent it may help to increase (.+?) which will split the rewrite operation into smaller commits\\.","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java","lineNumber":376,"sourceCode":"      LOG.warn(\n          \"{} is true but {} rewrite commits failed. Check the logs to determine why the individual \"\n              + \"commits failed. If this is persistent it may help to increase {} which will split the rewrite operation \"\n              + \"into smaller commits.\",\n          PARTIAL_PROGRESS_ENABLED,\n          failedCommits,\n          PARTIAL_PROGRESS_MAX_COMMITS);\n    } else if (failedCommits > maxFailedCommits) {\n      String errorMessage =\n          String.format(\n              Locale.ROOT,\n              \"%s is true but %d rewrite commits failed. This is more than the maximum allowed failures of %d. \"\n                  + \"Check the logs to determine why the individual commits failed. If this is persistent it may help to \"\n                  + \"increase %s which will split the rewrite operation into smaller commits.\",\n              PARTIAL_PROGRESS_ENABLED,\n              failedCommits,\n              maxFailedCommits,\n              PARTIAL_PROGRESS_MAX_COMMITS);\n      throw new RuntimeException(errorMessage);\n    }\n\n    return ImmutableRewriteDataFiles.Result.builder()\n        .rewriteResults(toRewriteResults(commitService.results()))\n        .rewriteFailures(rewriteFailures);\n  }\n\n  private Iterable<FileGroupRewriteResult> toRewriteResults(List<RewriteFileGroup> commitResults) {\n    return commitResults.stream().map(RewriteFileGroup::asResult).collect(Collectors.toList());\n  }\n\n  void validateAndInitOptions() {\n    Set<String> validOptions = Sets.newHashSet(runner.validOptions());\n    validOptions.addAll(VALID_OPTIONS);\n    validOptions.addAll(planner.validOptions());\n\n    Set<String> invalidKeys = Sets.newHashSet(options().keySet());\n    invalidKeys.removeAll(validOptions);","sourceCodeStart":358,"sourceCodeEnd":394,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java#L358-L394","documentation":"In doExecuteWithPartialProgress, when partial progress is enabled but the number of failed group commits exceeds the maximum allowed failures (computed from max commits), the action throws RuntimeException telling you to inspect commit failure logs and possibly increase max commits.","triggerScenarios":"Running rewriteDataFiles with rewrite.partial-progress.enabled=true where failedCommits exceeds (maxCommits - commitsSucceeded) budget — i.e. more than the allowed number of group commits failed (persistent conflicts, oversized files, catalog failures).","commonSituations":"Heavy concurrent write traffic causing most group commits to conflict; too few allowed commits (small max-commits value) relative to group count; persistent table metadata issues or catalog outages.","solutions":["Increase rewrite.partial-progress.max-commits so more failures are tolerated and commits are smaller","Check executor/driver logs for the underlying CommitFailedException causes and fix them (concurrent writers, catalog throttling)","Reduce concurrency or reschedule the rewrite during quieter periods and re-run","If conflicts dominate, consider non-partial mode with retries or rewrite smaller data sizes per group"],"exampleFix":"// before\n.option(\"rewrite.partial-progress.enabled\", \"true\")\n.option(\"rewrite.partial-progress.max-commits\", \"2\")\n// after\n.option(\"rewrite.partial-progress.enabled\", \"true\")\n.option(\"rewrite.partial-progress.max-commits\", \"20\")","handlingStrategy":"retry","validationCode":"int groups = estimateRewriteGroupCount(table, options);\nif (groups > maxCommits) { increaseMaxCommits(groups); }","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Set max-commits comfortably above expected group count","Fix persistent commit failures seen in logs before re-running","Reduce concurrent writers during partial-progress rewrites","Keep per-group data volume small"],"tags":["spark","rewrite-data-files","partial-progress","commit-failure"],"backgroundTag":"http-error-response","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}