{"record":{"id":"6f2786387d782635","repo":"apache/iceberg","slug":"s-is-true-but-d-rewrite-commits-failed-this-is-6f2786","errorCode":null,"errorMessage":"%s is true but %d rewrite commits failed. This is more than the maximum allowed failures of %d. Check the logs to determine why the individual commits failed. If this is persistent it may help to increase %s which will split the rewrite operation into smaller commits.","messagePattern":"(.+?) is true but (.+?) rewrite commits failed\\. This is more than the maximum allowed failures of (.+?)\\. Check the logs to determine why the individual commits failed\\. If this is persistent it may help to increase (.+?) which will split the rewrite operation into smaller commits\\.","errorType":"exception","errorClass":"java.lang.RuntimeException","httpStatus":null,"severity":"error","filePath":"spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java","lineNumber":383,"sourceCode":"      LOG.warn(\n          \"{} is true but {} rewrite commits failed. Check the logs to determine why the individual \"\n              + \"commits failed. If this is persistent it may help to increase {} which will split the rewrite operation \"\n              + \"into smaller commits.\",\n          PARTIAL_PROGRESS_ENABLED,\n          failedCommits,\n          PARTIAL_PROGRESS_MAX_COMMITS);\n    } else if (failedCommits > maxFailedCommits) {\n      String errorMessage =\n          String.format(\n              Locale.ROOT,\n              \"%s is true but %d rewrite commits failed. This is more than the maximum allowed failures of %d. \"\n                  + \"Check the logs to determine why the individual commits failed. If this is persistent it may help to \"\n                  + \"increase %s which will split the rewrite operation into smaller commits.\",\n              PARTIAL_PROGRESS_ENABLED,\n              failedCommits,\n              maxFailedCommits,\n              PARTIAL_PROGRESS_MAX_COMMITS);\n      throw new RuntimeException(errorMessage);\n    }\n\n    return ImmutableRewriteDataFiles.Result.builder()\n        .rewriteResults(toRewriteResults(commitService.results()))\n        .rewriteFailures(rewriteFailures);\n  }\n\n  private Iterable<FileGroupRewriteResult> toRewriteResults(List<RewriteFileGroup> commitResults) {\n    return commitResults.stream().map(RewriteFileGroup::asResult).collect(Collectors.toList());\n  }\n\n  void validateAndInitOptions() {\n    Set<String> validOptions = Sets.newHashSet(runner.validOptions());\n    validOptions.addAll(VALID_OPTIONS);\n    validOptions.addAll(planner.validOptions());\n\n    Set<String> invalidKeys = Sets.newHashSet(options().keySet());\n    invalidKeys.removeAll(validOptions);","sourceCodeStart":365,"sourceCodeEnd":401,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java#L365-L401","documentation":"In partial-progress mode, RewriteDataFiles tolerates up to (max commits - ... allowed failures) failed commits. When the number of failed rewrite commits exceeds partial-progress.max-failed-commits (derived from max_commits), the action throws this RuntimeException summarizing how many commits failed and how to tune the split.","triggerScenarios":"Running rewrite_data_files with partial_progress.enabled=true where more commits fail than the configured maximum — usually persistent conflicts with concurrent operations or repeated validation failures on the rewritten file groups.","commonSituations":"Heavy concurrent writer activity making most partial commits fail; too-small max_commits value allowing too few failures; systematic validation errors (e.g. schema/sequence conflicts) affecting many groups.","solutions":["Increase partial-progress.max-commits (or max_failed_commits) so more individual failures are tolerated","Eliminate the source of commit conflicts: pause concurrent maintenance jobs or coordinate rewrite scheduling","Inspect logs for the per-commit failure reason; fix systematic validation errors before re-running","Retry the rewrite when the table is quiescent"],"exampleFix":"// before\nCALL iceberg.system.rewrite_data_files(\n  table => 'db.t',\n  options => map('partial-progress.enabled', 'true', 'partial-progress.max-commits', '1'))\n-- after\nCALL iceberg.system.rewrite_data_files(\n  table => 'db.t',\n  options => map('partial-progress.enabled', 'true', 'partial-progress.max-commits', '10'));","handlingStrategy":"retry","validationCode":"// Sanity-check the partial progress options before running\n// partial-progress.enabled=true, partial-progress.max-commits >= 1","typeGuard":null,"tryCatchPattern":"try { rewriteAction.option(\"partial-progress.enabled\", \"true\").execute(); } catch (RuntimeException e) { if (e.getMessage().contains(\"rewrite commits failed\")) { /* increase partial-progress.max-commits and retry when table is quiescent */ } }","preventionTips":["Size partial-progress.max-commits generously relative to expected conflicts","Eliminate persistent commit conflicts (concurrent writers, streaming ingest) before re-running","Read the per-commit failure logs; repeated identical failures indicate a systematic issue, not tuning"],"tags":["spark","compaction","partial-progress","commit-conflict","concurrency"],"backgroundTag":"invalid-config-value","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}