apache/iceberg · error · RuntimeException

%s is true but %d rewrite commits failed. This is more than

Error message

%s is true but %d rewrite commits failed. This is more than the maximum allowed failures of %d. Check the logs to determine why the individual commits failed. If this is persistent it may help to increase %s which will split the rewrite operation into smaller commits.

What it means

In doExecuteWithPartialProgress, when partial progress is enabled but the number of failed group commits exceeds the maximum allowed failures (computed from max commits), the action throws RuntimeException telling you to inspect commit failure logs and possibly increase max commits.

Source

Thrown at spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java:376

      LOG.warn(
          "{} is true but {} rewrite commits failed. Check the logs to determine why the individual "
              + "commits failed. If this is persistent it may help to increase {} which will split the rewrite operation "
              + "into smaller commits.",
          PARTIAL_PROGRESS_ENABLED,
          failedCommits,
          PARTIAL_PROGRESS_MAX_COMMITS);
    } else if (failedCommits > maxFailedCommits) {
      String errorMessage =
          String.format(
              Locale.ROOT,
              "%s is true but %d rewrite commits failed. This is more than the maximum allowed failures of %d. "
                  + "Check the logs to determine why the individual commits failed. If this is persistent it may help to "
                  + "increase %s which will split the rewrite operation into smaller commits.",
              PARTIAL_PROGRESS_ENABLED,
              failedCommits,
              maxFailedCommits,
              PARTIAL_PROGRESS_MAX_COMMITS);
      throw new RuntimeException(errorMessage);
    }

    return ImmutableRewriteDataFiles.Result.builder()
        .rewriteResults(toRewriteResults(commitService.results()))
        .rewriteFailures(rewriteFailures);
  }

  private Iterable<FileGroupRewriteResult> toRewriteResults(List<RewriteFileGroup> commitResults) {
    return commitResults.stream().map(RewriteFileGroup::asResult).collect(Collectors.toList());
  }

  void validateAndInitOptions() {
    Set<String> validOptions = Sets.newHashSet(runner.validOptions());
    validOptions.addAll(VALID_OPTIONS);
    validOptions.addAll(planner.validOptions());

    Set<String> invalidKeys = Sets.newHashSet(options().keySet());
    invalidKeys.removeAll(validOptions);

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Increase rewrite.partial-progress.max-commits so more failures are tolerated and commits are smaller
2. Check executor/driver logs for the underlying CommitFailedException causes and fix them (concurrent writers, catalog throttling)
3. Reduce concurrency or reschedule the rewrite during quieter periods and re-run
4. If conflicts dominate, consider non-partial mode with retries or rewrite smaller data sizes per group

Example fix

// before
.option("rewrite.partial-progress.enabled", "true")
.option("rewrite.partial-progress.max-commits", "2")
// after
.option("rewrite.partial-progress.enabled", "true")
.option("rewrite.partial-progress.max-commits", "20")

int groups = estimateRewriteGroupCount(table, options);
if (groups > maxCommits) { increaseMaxCommits(groups); }

When it happens

Trigger: Running rewriteDataFiles with rewrite.partial-progress.enabled=true where failedCommits exceeds (maxCommits - commitsSucceeded) budget — i.e. more than the allowed number of group commits failed (persistent conflicts, oversized files, catalog failures).

Common situations: Heavy concurrent write traffic causing most group commits to conflict; too few allowed commits (small max-commits value) relative to group count; persistent table metadata issues or catalog outages.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

• %s is true but %d rewrite commits failed. This is more than
• %s is true but %d rewrite commits failed. This is more than
• {} is true but {} rewrite commits failed. Check the logs to
• Cannot commit rewrite because of a ValidationException or Co
• Cannot mix identity sort columns and a Zorder sort expressio

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/06fb268430edaf0e. Report an issue: GitHub.