{} is true but {} rewrite commits failed. Check the logs to

Error message

{} is true but {} rewrite commits failed. Check the logs to determine why the individual commits failed. If this is persistent it may help to increase {} which will split the rewrite operation into smaller commits.

What it means

A logged warning emitted by RewriteDataFilesSparkAction.doExecuteWithPartialProgress when partial-progress mode is enabled but some rewrite group commits failed while staying within the allowed maxFailedCommits. The overall action still succeeds with the groups that committed; this warns that compaction was incomplete and suggests raising partial-progress.max-commits so rewrites split into smaller commits.

Source

Thrown at spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java:365

        .onFailure(
            (fileGroup, exception) -> {
              LOG.error("Failure during rewrite group {}", fileGroup.info(), exception);
              rewriteFailures.add(
                  ImmutableRewriteDataFiles.FileGroupFailureResult.builder()
                      .info(fileGroup.info())
                      .dataFilesCount(fileGroup.inputFileNum())
                      .build());
            })
        .run(fileGroup -> commitService.offer(rewriteFiles(plan, fileGroup)));
    rewriteService.shutdown();

    // stop commit service
    commitService.close();

    int totalCommits = Math.min(plan.totalGroupCount(), maxCommits);
    int failedCommits = totalCommits - commitService.succeededCommits();
    if (failedCommits > 0 && failedCommits <= maxFailedCommits) {
      LOG.warn(
          "{} is true but {} rewrite commits failed. Check the logs to determine why the individual "
              + "commits failed. If this is persistent it may help to increase {} which will split the rewrite operation "
              + "into smaller commits.",
          PARTIAL_PROGRESS_ENABLED,
          failedCommits,
          PARTIAL_PROGRESS_MAX_COMMITS);
    } else if (failedCommits > maxFailedCommits) {
      String errorMessage =
          String.format(
              Locale.ROOT,
              "%s is true but %d rewrite commits failed. This is more than the maximum allowed failures of %d. "
                  + "Check the logs to determine why the individual commits failed. If this is persistent it may help to "
                  + "increase %s which will split the rewrite operation into smaller commits.",
              PARTIAL_PROGRESS_ENABLED,
              failedCommits,
              maxFailedCommits,
              PARTIAL_PROGRESS_MAX_COMMITS);
      throw new RuntimeException(errorMessage);

View on GitHub (pinned to 86d9c8fc54)

Solutions

Increase partial-progress.max-commits so each commit contains fewer groups and conflicts less.
Retry the rewrite; only groups whose commits failed remain un-compacted.
Reduce concurrent writers during compaction, or serialize maintenance jobs.
Check individual commit failure logs (validation failures, catalog errors) and address the root cause.
If conflicts persist, lower the group count or run rewrites during quiescent windows.

Example fix

// before
rewrite_data_files(partial_progress.enabled = true, partial_progress.max_commits = 1)
// after
rewrite_data_files(
  partial_progress.enabled = true,
  partial_progress.max_commits = 10  // smaller commits, fewer conflicts
)

Defensive patterns

Strategy: retry

Validate before calling

// check table-level commit contention before running partial-progress rewrites
Table table = catalog.loadTable(TableIdentifier.of("db", "tbl"));
int recentSnapshots = java.util.Lists.newArrayList(table.snapshots()).size(); // ensure no heavy concurrent writers

Try / catch

try { rewrite.execute(); } catch (Exception e) { /* commits within maxFailedCommits are tolerated; re-run to finish */ }

Prevention

Raise partial-progress.max-commits to shrink per-commit conflict surface
Schedule rewrites during quiescent windows without concurrent writers
Investigate individual commit failure logs first
Serialize maintenance jobs on the same table

When it happens

Trigger: Running rewrite_data_files with partial_progress.enabled=true when concurrent commits (e.g., from other jobs) cause RewriteFileGroup commit failures — CommitStateUnknownException, validation failures against a moved base table snapshot, or catalog conflicts — but failedCommits <= partial-progress.max-commits.

Common situations: Concurrent writes/compaction on the same table causing commit conflicts; REST catalog rate-limiting or flakiness during commit bursts; too few max-commits so each commit carries too many groups and conflicts more often; long-running rewrites while the table's latest snapshot advances.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/d4534be54512caa2. Report an issue: GitHub.