apache/iceberg · error · RuntimeException

%s is true but %d rewrite commits failed. This is more than

Error message

%s is true but %d rewrite commits failed. This is more than the maximum allowed failures of %d. Check the logs to determine why the individual commits failed. If this is persistent it may help to increase %s which will split the rewrite operation into smaller commits.

What it means

In doExecuteWithPartialProgress, rewrite groups are committed individually and some commits failed. When the number of failed commits exceeds PARTIAL_PROGRESS_MAX_COMMITS, the action aborts with this RuntimeException instead of continuing, reporting the failure count and the cap.

Source

Thrown at spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java:376

      LOG.warn(
          "{} is true but {} rewrite commits failed. Check the logs to determine why the individual "
              + "commits failed. If this is persistent it may help to increase {} which will split the rewrite operation "
              + "into smaller commits.",
          PARTIAL_PROGRESS_ENABLED,
          failedCommits,
          PARTIAL_PROGRESS_MAX_COMMITS);
    } else if (failedCommits > maxFailedCommits) {
      String errorMessage =
          String.format(
              Locale.ROOT,
              "%s is true but %d rewrite commits failed. This is more than the maximum allowed failures of %d. "
                  + "Check the logs to determine why the individual commits failed. If this is persistent it may help to "
                  + "increase %s which will split the rewrite operation into smaller commits.",
              PARTIAL_PROGRESS_ENABLED,
              failedCommits,
              maxFailedCommits,
              PARTIAL_PROGRESS_MAX_COMMITS);
      throw new RuntimeException(errorMessage);
    }

    return ImmutableRewriteDataFiles.Result.builder()
        .rewriteResults(toRewriteResults(commitService.results()))
        .rewriteFailures(rewriteFailures);
  }

  private Iterable<FileGroupRewriteResult> toRewriteResults(List<RewriteFileGroup> commitResults) {
    return commitResults.stream().map(RewriteFileGroup::asResult).collect(Collectors.toList());
  }

  void validateAndInitOptions() {
    Set<String> validOptions = Sets.newHashSet(runner.validOptions());
    validOptions.addAll(VALID_OPTIONS);
    validOptions.addAll(planner.validOptions());

    Set<String> invalidKeys = Sets.newHashSet(options().keySet());
    invalidKeys.removeAll(validOptions);

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Increase partial-progress.max-commits to tolerate more failed group commits (e.g. set to a fraction of total groups).
2. Pause or throttle concurrent writers during the rewrite to reduce commit conflicts.
3. Re-run the rewrite after load decreases; successful groups are already committed and only remaining groups will be rewritten.
4. Review logs/rewriteFailures for the underlying per-commit errors in case a non-conflict bug (e.g. validation) is the real cause.

Example fix

// before
.option("partial-progress.enabled", "true")
.option("partial-progress.max-commits", "1")
// after
.option("partial-progress.enabled", "true")
.option("partial-progress.max-commits", "10")

// Budget: ensure max-commits tolerates expected conflict rate
int groups = estimateGroups(table);
int maxCommits = Integer.parseInt(table.properties()
    .getOrDefault(TableProperties.PARTIAL_PROGRESS_MAX_COMMITS, "10"));
if (maxCommits < groups / 2) LOG.warn("max-commits too low for busy table; raise partial-progress.max-commits");

try {
  rewrite.execute();
} catch (RuntimeException e) {
  if (e.getMessage().contains("maximum allowed failures")) {
    // raise partial-progress.max-commits and/or re-run during a quiet window
  } else throw e;
}

When it happens

Trigger: Running rewriteDataFiles with partial-progress.enabled=true where failedCommits > partial-progress.max-commits (default 1) — i.e. multiple group commits hit CommitFailedException/ValidationException, typically under heavy concurrent writing.

Common situations: Compacting a very busy streaming table where every commit races with writers; too many groups competing for commits with a low max-commits budget; sustained table churn during the rewrite.

Related errors

• %s is true but %d rewrite commits failed. This is more than
• %s is true but %d rewrite commits failed. This is more than
• {} is true but {} rewrite commits failed. Check the logs to
• Cannot commit rewrite because of a ValidationException or Co
• Cannot mix identity sort columns and a Zorder sort expressio

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/250fe6df284e1c35. Report an issue: GitHub.