apache/iceberg · error · java.lang.RuntimeException

%s is true but %d rewrite commits failed. This is more than

Error message

%s is true but %d rewrite commits failed. This is more than the maximum allowed failures of %d. Check the logs to determine why the individual commits failed. If this is persistent it may help to increase %s which will split the rewrite operation into smaller commits.

What it means

In partial-progress mode, RewriteDataFiles tolerates up to (max commits - ... allowed failures) failed commits. When the number of failed rewrite commits exceeds partial-progress.max-failed-commits (derived from max_commits), the action throws this RuntimeException summarizing how many commits failed and how to tune the split.

Source

Thrown at spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java:383

      LOG.warn(
          "{} is true but {} rewrite commits failed. Check the logs to determine why the individual "
              + "commits failed. If this is persistent it may help to increase {} which will split the rewrite operation "
              + "into smaller commits.",
          PARTIAL_PROGRESS_ENABLED,
          failedCommits,
          PARTIAL_PROGRESS_MAX_COMMITS);
    } else if (failedCommits > maxFailedCommits) {
      String errorMessage =
          String.format(
              Locale.ROOT,
              "%s is true but %d rewrite commits failed. This is more than the maximum allowed failures of %d. "
                  + "Check the logs to determine why the individual commits failed. If this is persistent it may help to "
                  + "increase %s which will split the rewrite operation into smaller commits.",
              PARTIAL_PROGRESS_ENABLED,
              failedCommits,
              maxFailedCommits,
              PARTIAL_PROGRESS_MAX_COMMITS);
      throw new RuntimeException(errorMessage);
    }

    return ImmutableRewriteDataFiles.Result.builder()
        .rewriteResults(toRewriteResults(commitService.results()))
        .rewriteFailures(rewriteFailures);
  }

  private Iterable<FileGroupRewriteResult> toRewriteResults(List<RewriteFileGroup> commitResults) {
    return commitResults.stream().map(RewriteFileGroup::asResult).collect(Collectors.toList());
  }

  void validateAndInitOptions() {
    Set<String> validOptions = Sets.newHashSet(runner.validOptions());
    validOptions.addAll(VALID_OPTIONS);
    validOptions.addAll(planner.validOptions());

    Set<String> invalidKeys = Sets.newHashSet(options().keySet());
    invalidKeys.removeAll(validOptions);

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Increase partial-progress.max-commits (or max_failed_commits) so more individual failures are tolerated
2. Eliminate the source of commit conflicts: pause concurrent maintenance jobs or coordinate rewrite scheduling
3. Inspect logs for the per-commit failure reason; fix systematic validation errors before re-running
4. Retry the rewrite when the table is quiescent

Example fix

// before
CALL iceberg.system.rewrite_data_files(
  table => 'db.t',
  options => map('partial-progress.enabled', 'true', 'partial-progress.max-commits', '1'))
-- after
CALL iceberg.system.rewrite_data_files(
  table => 'db.t',
  options => map('partial-progress.enabled', 'true', 'partial-progress.max-commits', '10'));

// Sanity-check the partial progress options before running
// partial-progress.enabled=true, partial-progress.max-commits >= 1

try { rewriteAction.option("partial-progress.enabled", "true").execute(); } catch (RuntimeException e) { if (e.getMessage().contains("rewrite commits failed")) { /* increase partial-progress.max-commits and retry when table is quiescent */ } }

When it happens

Trigger: Running rewrite_data_files with partial_progress.enabled=true where more commits fail than the configured maximum — usually persistent conflicts with concurrent operations or repeated validation failures on the rewritten file groups.

Common situations: Heavy concurrent writer activity making most partial commits fail; too-small max_commits value allowing too few failures; systematic validation errors (e.g. schema/sequence conflicts) affecting many groups.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

• Cannot commit rewrite because of a ValidationException or Co
• {} is true but {} rewrite commits failed. Check the logs to
• {} is true but {} rewrite commits failed. Check the logs to
• Cannot commit rewrite because of a ValidationException or Co
• Cannot commit rewrite because of a ValidationException or Co

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/6f2786387d782635. Report an issue: GitHub.