apache/iceberg · error · RuntimeException

%s is true but %d rewrite commits failed. This is more than

Error message

%s is true but %d rewrite commits failed. This is more than the maximum allowed failures of %d. Check the logs to determine why the individual commits failed. If this is persistent it may help to increase %s which will split the rewrite operation into smaller commits.

What it means

With partial progress enabled, RewriteDataFiles tolerates failed commits up to maxFailedCommits derived from PARTIAL_PROGRESS_MAX_COMMITS. When the number of failed commits exceeds that allowance, the action fails with a RuntimeException summarizing the failures and suggesting a larger commit count.

Source

Thrown at spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java:383

      LOG.warn(
          "{} is true but {} rewrite commits failed. Check the logs to determine why the individual "
              + "commits failed. If this is persistent it may help to increase {} which will split the rewrite operation "
              + "into smaller commits.",
          PARTIAL_PROGRESS_ENABLED,
          failedCommits,
          PARTIAL_PROGRESS_MAX_COMMITS);
    } else if (failedCommits > maxFailedCommits) {
      String errorMessage =
          String.format(
              Locale.ROOT,
              "%s is true but %d rewrite commits failed. This is more than the maximum allowed failures of %d. "
                  + "Check the logs to determine why the individual commits failed. If this is persistent it may help to "
                  + "increase %s which will split the rewrite operation into smaller commits.",
              PARTIAL_PROGRESS_ENABLED,
              failedCommits,
              maxFailedCommits,
              PARTIAL_PROGRESS_MAX_COMMITS);
      throw new RuntimeException(errorMessage);
    }

    return ImmutableRewriteDataFiles.Result.builder()
        .rewriteResults(toRewriteResults(commitService.results()))
        .rewriteFailures(rewriteFailures);
  }

  private Iterable<FileGroupRewriteResult> toRewriteResults(List<RewriteFileGroup> commitResults) {
    return commitResults.stream().map(RewriteFileGroup::asResult).collect(Collectors.toList());
  }

  void validateAndInitOptions() {
    Set<String> validOptions = Sets.newHashSet(runner.validOptions());
    validOptions.addAll(VALID_OPTIONS);
    validOptions.addAll(planner.validOptions());

    Set<String> invalidKeys = Sets.newHashSet(options().keySet());
    invalidKeys.removeAll(validOptions);

View on GitHub (pinned to 86d9c8fc54)

Solutions

Increase partial-progress.max-commits so the allowed failure budget grows with the number of commits
Reduce concurrent operations competing for commits on the table
Inspect logs for the root cause of individual commit failures and fix it (e.g. conflicting requirement, bad group), then re-run

Example fix

// before
.option("partial-progress.enabled", "true")
.option("partial-progress.max-commits", "2") // too few for contended table
// after
.option("partial-progress.enabled", "true")
.option("partial-progress.max-commits", "20")

Defensive patterns

Strategy: retry

Validate before calling

// ensure failure budget fits expected contention
int maxCommits = conf.get(...PARTIAL_PROGRESS_MAX_COMMITS...);
int allowedFailures = maxCommits / 10; // per action's formula
if (expectedConcurrentWriters > allowedFailures) { increaseMaxCommits(); }

Try / catch

try { action.execute(); } catch (RuntimeException e) { if (e.getMessage().contains("maximum allowed failures")) { /* raise partial-progress.max-commits and retry */ } }

Prevention

Set partial-progress.max-commits generously on contended tables
Investigate root causes of individual commit failures in logs
Reduce writer concurrency during compaction windows

When it happens

Trigger: doExecuteWithPartialProgress observes failedCommits exceeding the allowed maximum — i.e. more than maxFailedCommits group commits failed, typically from repeated CommitFailedException due to heavy concurrency.

Common situations: Sustained concurrent writer contention exceeding the allowed failure budget; a systematically failing commit group (bad data files, validation conflict) retrying until the budget is exhausted.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/0cc19ba957483b49. Report an issue: GitHub.