{"record":{"id":"0cc19ba957483b49","repo":"apache/iceberg","slug":"s-is-true-but-d-rewrite-commits-failed-this-is-0cc19b","errorCode":null,"errorMessage":"%s is true but %d rewrite commits failed. This is more than the maximum allowed failures of %d. Check the logs to determine why the individual commits failed. If this is persistent it may help to increase %s which will split the rewrite operation into smaller commits.","messagePattern":"(.+?) is true but (.+?) rewrite commits failed\\. This is more than the maximum allowed failures of (.+?)\\. Check the logs to determine why the individual commits failed\\. If this is persistent it may help to increase (.+?) which will split the rewrite operation into smaller commits\\.","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java","lineNumber":383,"sourceCode":"      LOG.warn(\n          \"{} is true but {} rewrite commits failed. Check the logs to determine why the individual \"\n              + \"commits failed. If this is persistent it may help to increase {} which will split the rewrite operation \"\n              + \"into smaller commits.\",\n          PARTIAL_PROGRESS_ENABLED,\n          failedCommits,\n          PARTIAL_PROGRESS_MAX_COMMITS);\n    } else if (failedCommits > maxFailedCommits) {\n      String errorMessage =\n          String.format(\n              Locale.ROOT,\n              \"%s is true but %d rewrite commits failed. This is more than the maximum allowed failures of %d. \"\n                  + \"Check the logs to determine why the individual commits failed. If this is persistent it may help to \"\n                  + \"increase %s which will split the rewrite operation into smaller commits.\",\n              PARTIAL_PROGRESS_ENABLED,\n              failedCommits,\n              maxFailedCommits,\n              PARTIAL_PROGRESS_MAX_COMMITS);\n      throw new RuntimeException(errorMessage);\n    }\n\n    return ImmutableRewriteDataFiles.Result.builder()\n        .rewriteResults(toRewriteResults(commitService.results()))\n        .rewriteFailures(rewriteFailures);\n  }\n\n  private Iterable<FileGroupRewriteResult> toRewriteResults(List<RewriteFileGroup> commitResults) {\n    return commitResults.stream().map(RewriteFileGroup::asResult).collect(Collectors.toList());\n  }\n\n  void validateAndInitOptions() {\n    Set<String> validOptions = Sets.newHashSet(runner.validOptions());\n    validOptions.addAll(VALID_OPTIONS);\n    validOptions.addAll(planner.validOptions());\n\n    Set<String> invalidKeys = Sets.newHashSet(options().keySet());\n    invalidKeys.removeAll(validOptions);","sourceCodeStart":365,"sourceCodeEnd":401,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java#L365-L401","documentation":"With partial progress enabled, RewriteDataFiles tolerates failed commits up to maxFailedCommits derived from PARTIAL_PROGRESS_MAX_COMMITS. When the number of failed commits exceeds that allowance, the action fails with a RuntimeException summarizing the failures and suggesting a larger commit count.","triggerScenarios":"doExecuteWithPartialProgress observes failedCommits exceeding the allowed maximum — i.e. more than maxFailedCommits group commits failed, typically from repeated CommitFailedException due to heavy concurrency.","commonSituations":"Sustained concurrent writer contention exceeding the allowed failure budget; a systematically failing commit group (bad data files, validation conflict) retrying until the budget is exhausted.","solutions":["Increase partial-progress.max-commits so the allowed failure budget grows with the number of commits","Reduce concurrent operations competing for commits on the table","Inspect logs for the root cause of individual commit failures and fix it (e.g. conflicting requirement, bad group), then re-run"],"exampleFix":"// before\n.option(\"partial-progress.enabled\", \"true\")\n.option(\"partial-progress.max-commits\", \"2\") // too few for contended table\n// after\n.option(\"partial-progress.enabled\", \"true\")\n.option(\"partial-progress.max-commits\", \"20\")","handlingStrategy":"retry","validationCode":"// ensure failure budget fits expected contention\nint maxCommits = conf.get(...PARTIAL_PROGRESS_MAX_COMMITS...);\nint allowedFailures = maxCommits / 10; // per action's formula\nif (expectedConcurrentWriters > allowedFailures) { increaseMaxCommits(); }","typeGuard":null,"tryCatchPattern":"try { action.execute(); } catch (RuntimeException e) { if (e.getMessage().contains(\"maximum allowed failures\")) { /* raise partial-progress.max-commits and retry */ } }","preventionTips":["Set partial-progress.max-commits generously on contended tables","Investigate root causes of individual commit failures in logs","Reduce writer concurrency during compaction windows"],"tags":["spark","iceberg","partial-progress","commit-failure"],"backgroundTag":"rate-limit-exceeded","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}