{} is true but {} rewrite commits failed. Check the logs to

Error message

{} is true but {} rewrite commits failed. Check the logs to determine why the individual commits failed. If this is persistent it may help to increase {} which will split the rewrite operation into smaller commits.

What it means

RewriteDataFilesSparkAction with partial progress enabled logs this warning when some rewrite group commits failed but stayed within max-failed-commits, so the action completes with partial results instead of failing. The library logs it because individual group commits can fail (e.g. concurrent table updates) while the overall operation still makes progress.

Source

Thrown at spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java:365

        .onFailure(
            (fileGroup, exception) -> {
              LOG.error("Failure during rewrite group {}", fileGroup.info(), exception);
              rewriteFailures.add(
                  ImmutableRewriteDataFiles.FileGroupFailureResult.builder()
                      .info(fileGroup.info())
                      .dataFilesCount(fileGroup.inputFileNum())
                      .build());
            })
        .run(fileGroup -> commitService.offer(rewriteFiles(plan, fileGroup)));
    rewriteService.shutdown();

    // stop commit service
    commitService.close();

    int totalCommits = Math.min(plan.totalGroupCount(), maxCommits);
    int failedCommits = totalCommits - commitService.succeededCommits();
    if (failedCommits > 0 && failedCommits <= maxFailedCommits) {
      LOG.warn(
          "{} is true but {} rewrite commits failed. Check the logs to determine why the individual "
              + "commits failed. If this is persistent it may help to increase {} which will split the rewrite operation "
              + "into smaller commits.",
          PARTIAL_PROGRESS_ENABLED,
          failedCommits,
          PARTIAL_PROGRESS_MAX_COMMITS);
    } else if (failedCommits > maxFailedCommits) {
      String errorMessage =
          String.format(
              Locale.ROOT,
              "%s is true but %d rewrite commits failed. This is more than the maximum allowed failures of %d. "
                  + "Check the logs to determine why the individual commits failed. If this is persistent it may help to "
                  + "increase %s which will split the rewrite operation into smaller commits.",
              PARTIAL_PROGRESS_ENABLED,
              failedCommits,
              maxFailedCommits,
              PARTIAL_PROGRESS_MAX_COMMITS);
      throw new RuntimeException(errorMessage);

View on GitHub (pinned to 86d9c8fc54)

Solutions

Inspect executor/driver logs for the per-group commit exceptions
Increase partial-progress.max-commits to split work into smaller commits
Reduce concurrency with other writers or retry the rewrite after the conflicting commits finish
If failures are persistent and exceeded max-failed-commits, expect the action to fail; fix the underlying commit conflict first

Example fix

// before
SparkActions.get(spark).rewriteDataFiles(table).option("partial-progress.enabled", "true").execute();
// after
SparkActions.get(spark).rewriteDataFiles(table)
    .option("partial-progress.enabled", "true")
    .option("partial-progress.max-commits", "20")
    .execute();

Defensive patterns

Strategy: validation

Validate before calling

boolean partial = Boolean.parseBoolean(table.properties().getOrDefault("write.spark.partial-progress.enabled", "false"));
int maxCommits = Integer.parseInt(table.properties().getOrDefault("write.spark.partial-progress.max-commits", "10"));
if (partial && hasConcurrentWriters(table)) { /* schedule rewrite in a window with no other writers */ }

Prevention

Avoid running rewrites concurrently with streaming/other batch writers
Set partial-progress.max-commits proportional to table size
Monitor succeeded vs failed commit counts from action results
Check table snapshot history after rewrites to confirm expected commits

When it happens

Trigger: Calling RewriteDataFilesSparkAction.execute() with partial-progress.enabled=true where totalCommits - commitService.succeededCommits() > 0 and <= partial-progress.max-commits in doExecuteWithPartialProgress.

Common situations: Concurrent writers commit to the table between rewrite planning and commit causing ValidationException; commit contention with compaction jobs; expired snapshots during long rewrites.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/7958cf6a18958f11. Report an issue: GitHub.