{} is true but {} rewrite commits failed. Check the logs to

Error message

{} is true but {} rewrite commits failed. Check the logs to determine why the individual commits failed. If this is persistent it may help to increase {} which will split the rewrite operation into smaller commits.

What it means

With partial-progress enabled, RewriteDataFiles allows up to maxFailedCommits failed commits; when the number of failed commits falls within that allowance, the action logs this warning naming the partial-progress properties so users know some rewrite commits did not land but the job still succeeds.

Source

Thrown at spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java:358

        .onFailure(
            (fileGroup, exception) -> {
              LOG.error("Failure during rewrite group {}", fileGroup.info(), exception);
              rewriteFailures.add(
                  ImmutableRewriteDataFiles.FileGroupFailureResult.builder()
                      .info(fileGroup.info())
                      .dataFilesCount(fileGroup.inputFileNum())
                      .build());
            })
        .run(fileGroup -> commitService.offer(rewriteFiles(plan, fileGroup)));
    rewriteService.shutdown();

    // stop commit service
    commitService.close();

    int totalCommits = Math.min(plan.totalGroupCount(), maxCommits);
    int failedCommits = totalCommits - commitService.succeededCommits();
    if (failedCommits > 0 && failedCommits <= maxFailedCommits) {
      LOG.warn(
          "{} is true but {} rewrite commits failed. Check the logs to determine why the individual "
              + "commits failed. If this is persistent it may help to increase {} which will split the rewrite operation "
              + "into smaller commits.",
          PARTIAL_PROGRESS_ENABLED,
          failedCommits,
          PARTIAL_PROGRESS_MAX_COMMITS);
    } else if (failedCommits > maxFailedCommits) {
      String errorMessage =
          String.format(
              Locale.ROOT,
              "%s is true but %d rewrite commits failed. This is more than the maximum allowed failures of %d. "
                  + "Check the logs to determine why the individual commits failed. If this is persistent it may help to "
                  + "increase %s which will split the rewrite operation into smaller commits.",
              PARTIAL_PROGRESS_ENABLED,
              failedCommits,
              maxFailedCommits,
              PARTIAL_PROGRESS_MAX_COMMITS);
      throw new RuntimeException(errorMessage);

View on GitHub (pinned to 86d9c8fc54)

Solutions

Raise partial-progress.max-commits so each commit is smaller and less likely to conflict with concurrent writers.
Inspect earlier ERROR logs to find why individual commits failed (usually ValidationException from concurrent appends).
Reduce concurrent write load on the table or retry the rewrite when the table is quiet.
If failures exceed the allowance the job errors out — keeping partial-progress.enabled true limits wasted work either way.

Example fix

// before: too few, too-large commits that fail under concurrency
spark.sql("CALL cat.sys.rewrite_data_files(table => 'db.t', " +
  "options => map('partial-progress.enabled','true'))");
// after: split into more, smaller commits
spark.sql("CALL cat.sys.rewrite_data_files(table => 'db.t', " +
  "options => map('partial-progress.enabled','true','partial-progress.max-commits','50'))");

Defensive patterns

Strategy: retry

Validate before calling

int maxCommits = Integer.parseInt(props.getOrDefault("partial-progress.max-commits", "10"));
// ensure maxCommits is high relative to expected group count

Prevention

Set partial-progress.max-commits proportional to group count for large rewrites.
Avoid heavy concurrent writers during rewrite windows.
Retry the rewrite after transient commit conflicts; table state stays consistent.
Grep logs for ValidationException to identify conflicting operations.

When it happens

Trigger: Running RewriteDataFiles with partial-progress.enabled=true where (totalGroups capped at max-commits) minus commitService.succeededCommits() is > 0 and <= maxFailedCommits — i.e. one or more group commits failed but stayed within tolerance.

Common situations: Concurrent writers causing commit conflicts (CommitStateUnknown / validation failures); too few max commits for large rewrites forcing oversized commits that conflict; metastore contention.

Understand the failure class

Background: Conflicting config options: "cannot be used together" — configuration validation errors across open-source libraries — this error's family across 162 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/bf87009d69e24531. Report an issue: GitHub.