apache/iceberg · warning

{} is true but {} rewrite commits failed. Check the logs to

Error message

{} is true but {} rewrite commits failed. Check the logs to determine why the individual commits failed. If this is persistent it may help to increase {} which will split the rewrite operation into smaller commits.

What it means

WARN log emitted after a partial-progress rewrite finishes: partial-progress.enabled=true allows some commits to fail up to partial-progress.max-failed-commits, and the action reports that N of the total commits failed but the rewrite still completed partially. It is informational about degraded (not failed) results.

Source

Thrown at spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java:358

        .onFailure(
            (fileGroup, exception) -> {
              LOG.error("Failure during rewrite group {}", fileGroup.info(), exception);
              rewriteFailures.add(
                  ImmutableRewriteDataFiles.FileGroupFailureResult.builder()
                      .info(fileGroup.info())
                      .dataFilesCount(fileGroup.inputFileNum())
                      .build());
            })
        .run(fileGroup -> commitService.offer(rewriteFiles(plan, fileGroup)));
    rewriteService.shutdown();

    // stop commit service
    commitService.close();

    int totalCommits = Math.min(plan.totalGroupCount(), maxCommits);
    int failedCommits = totalCommits - commitService.succeededCommits();
    if (failedCommits > 0 && failedCommits <= maxFailedCommits) {
      LOG.warn(
          "{} is true but {} rewrite commits failed. Check the logs to determine why the individual "
              + "commits failed. If this is persistent it may help to increase {} which will split the rewrite operation "
              + "into smaller commits.",
          PARTIAL_PROGRESS_ENABLED,
          failedCommits,
          PARTIAL_PROGRESS_MAX_COMMITS);
    } else if (failedCommits > maxFailedCommits) {
      String errorMessage =
          String.format(
              Locale.ROOT,
              "%s is true but %d rewrite commits failed. This is more than the maximum allowed failures of %d. "
                  + "Check the logs to determine why the individual commits failed. If this is persistent it may help to "
                  + "increase %s which will split the rewrite operation into smaller commits.",
              PARTIAL_PROGRESS_ENABLED,
              failedCommits,
              maxFailedCommits,
              PARTIAL_PROGRESS_MAX_COMMITS);
      throw new RuntimeException(errorMessage);

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Check earlier WARN/error logs for each failed commit's reason.
2. Increase partial-progress.max-commits so each commit is smaller and less likely to conflict.
3. Re-run the rewrite to compact files missed by the failed commits.
4. Reduce concurrent write load or serialize compaction with other writers.

Example fix

// before
action.option("partial-progress.enabled", "true").execute();
// after: smaller commits reduce conflicts
action.option("partial-progress.enabled", "true")
      .option("partial-progress.max-commits", "50")
      .execute();

boolean partial = Boolean.parseBoolean(table.properties().getOrDefault("write.spark.partial-progress.enabled", "false")); int maxCommits = Integer.parseInt(table.properties().getOrDefault("write.spark.partial-progress.max-commits", "10"));

try { result = action.execute(); if (result.failedDataFilesCount() > 0) { /* log and schedule re-run */ } } catch (Exception e) { ... }

When it happens

Trigger: RewriteDataFilesSparkAction.doExecuteWithPartialProgress where commitService.succeededCommits() is less than totalCommits (min of plan.totalGroupCount() and maxCommits) and failedCommits <= maxFailedCommits — concurrent table updates invalidated some rewrite commits.

Common situations: Concurrent writers appending/overwriting snapshots while compaction commits, causing ValidateMetrics/commit conflicts; snapshot expiry removing needed snapshots; optimistic concurrency clashes under heavy load.

Related errors

• %s is true but %d rewrite commits failed. This is more than
• {} is true but {} rewrite commits failed. Check the logs to
• {} is true but {} rewrite commits failed. Check the logs to
• {} is true but {} rewrite commits failed. Check the logs to
• Cannot commit rewrite because of a ValidationException or Co

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/80d46607f007c907. Report an issue: GitHub.