{"record":{"id":"250fe6df284e1c35","repo":"apache/iceberg","slug":"s-is-true-but-d-rewrite-commits-failed-this-is","errorCode":null,"errorMessage":"%s is true but %d rewrite commits failed. This is more than the maximum allowed failures of %d. Check the logs to determine why the individual commits failed. If this is persistent it may help to increase %s which will split the rewrite operation into smaller commits.","messagePattern":"(.+?) is true but (.+?) rewrite commits failed\\. This is more than the maximum allowed failures of (.+?)\\. Check the logs to determine why the individual commits failed\\. If this is persistent it may help to increase (.+?) which will split the rewrite operation into smaller commits\\.","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java","lineNumber":376,"sourceCode":"      LOG.warn(\n          \"{} is true but {} rewrite commits failed. Check the logs to determine why the individual \"\n              + \"commits failed. If this is persistent it may help to increase {} which will split the rewrite operation \"\n              + \"into smaller commits.\",\n          PARTIAL_PROGRESS_ENABLED,\n          failedCommits,\n          PARTIAL_PROGRESS_MAX_COMMITS);\n    } else if (failedCommits > maxFailedCommits) {\n      String errorMessage =\n          String.format(\n              Locale.ROOT,\n              \"%s is true but %d rewrite commits failed. This is more than the maximum allowed failures of %d. \"\n                  + \"Check the logs to determine why the individual commits failed. If this is persistent it may help to \"\n                  + \"increase %s which will split the rewrite operation into smaller commits.\",\n              PARTIAL_PROGRESS_ENABLED,\n              failedCommits,\n              maxFailedCommits,\n              PARTIAL_PROGRESS_MAX_COMMITS);\n      throw new RuntimeException(errorMessage);\n    }\n\n    return ImmutableRewriteDataFiles.Result.builder()\n        .rewriteResults(toRewriteResults(commitService.results()))\n        .rewriteFailures(rewriteFailures);\n  }\n\n  private Iterable<FileGroupRewriteResult> toRewriteResults(List<RewriteFileGroup> commitResults) {\n    return commitResults.stream().map(RewriteFileGroup::asResult).collect(Collectors.toList());\n  }\n\n  void validateAndInitOptions() {\n    Set<String> validOptions = Sets.newHashSet(runner.validOptions());\n    validOptions.addAll(VALID_OPTIONS);\n    validOptions.addAll(planner.validOptions());\n\n    Set<String> invalidKeys = Sets.newHashSet(options().keySet());\n    invalidKeys.removeAll(validOptions);","sourceCodeStart":358,"sourceCodeEnd":394,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java#L358-L394","documentation":"In doExecuteWithPartialProgress, rewrite groups are committed individually and some commits failed. When the number of failed commits exceeds PARTIAL_PROGRESS_MAX_COMMITS, the action aborts with this RuntimeException instead of continuing, reporting the failure count and the cap.","triggerScenarios":"Running rewriteDataFiles with partial-progress.enabled=true where failedCommits > partial-progress.max-commits (default 1) — i.e. multiple group commits hit CommitFailedException/ValidationException, typically under heavy concurrent writing.","commonSituations":"Compacting a very busy streaming table where every commit races with writers; too many groups competing for commits with a low max-commits budget; sustained table churn during the rewrite.","solutions":["Increase partial-progress.max-commits to tolerate more failed group commits (e.g. set to a fraction of total groups).","Pause or throttle concurrent writers during the rewrite to reduce commit conflicts.","Re-run the rewrite after load decreases; successful groups are already committed and only remaining groups will be rewritten.","Review logs/rewriteFailures for the underlying per-commit errors in case a non-conflict bug (e.g. validation) is the real cause."],"exampleFix":"// before\n.option(\"partial-progress.enabled\", \"true\")\n.option(\"partial-progress.max-commits\", \"1\")\n// after\n.option(\"partial-progress.enabled\", \"true\")\n.option(\"partial-progress.max-commits\", \"10\")","handlingStrategy":"retry","validationCode":"// Budget: ensure max-commits tolerates expected conflict rate\nint groups = estimateGroups(table);\nint maxCommits = Integer.parseInt(table.properties()\n    .getOrDefault(TableProperties.PARTIAL_PROGRESS_MAX_COMMITS, \"10\"));\nif (maxCommits < groups / 2) LOG.warn(\"max-commits too low for busy table; raise partial-progress.max-commits\");","typeGuard":null,"tryCatchPattern":"try {\n  rewrite.execute();\n} catch (RuntimeException e) {\n  if (e.getMessage().contains(\"maximum allowed failures\")) {\n    // raise partial-progress.max-commits and/or re-run during a quiet window\n  } else throw e;\n}","preventionTips":["Set partial-progress.max-commits proportionally to the number of rewrite groups on busy tables.","Monitor rewriteFailures in the result to detect persistent commit problems early.","Throttle or pause concurrent writers during large rewrites."],"tags":["spark","rewrite-data-files","partial-progress","commit-failure"],"backgroundTag":"commit-conflict","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}