apache/iceberg · warning

Failure during rewrite process for group {}

Error message

Failure during rewrite process for group {}

What it means

WARN log from RewriteDataFilesSparkAction.doExecute: when a rewrite file group fails during its Spark job, the Tasks.foreach onFailure hook logs the group's RewriteGroupInfo and exception, then the action continues (with stopOnFailure stopping subsequent scheduling of this batch) rather than aborting the whole statement.

Source

Thrown at spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java:273

    return new RewriteDataFilesCommitManager(
        table, startingSnapshotId, useStartingSequenceNumber, commitSummary(), branch);
  }

  private Builder doExecute(
      FileRewritePlan<FileGroupInfo, FileScanTask, DataFile, RewriteFileGroup> plan,
      RewriteDataFilesCommitManager commitManager) {
    ExecutorService rewriteService = rewriteService();

    ConcurrentLinkedQueue<RewriteFileGroup> rewrittenGroups = new ConcurrentLinkedQueue<>();

    Tasks.Builder<RewriteFileGroup> rewriteTaskBuilder =
        Tasks.foreach(plan.groups())
            .executeWith(rewriteService)
            .stopOnFailure()
            .noRetry()
            .onFailure(
                (fileGroup, exception) -> {
                  LOG.warn(
                      "Failure during rewrite process for group {}", fileGroup.info(), exception);
                });

    try {
      rewriteTaskBuilder.run(
          fileGroup -> {
            rewrittenGroups.add(rewriteFiles(plan, fileGroup));
          });
    } catch (Exception e) {
      // At least one rewrite group failed, clean up all completed rewrites
      LOG.error(
          "Cannot complete rewrite, {} is not enabled and one of the file set groups failed to "
              + "be rewritten. This error occurred during the writing of new files, not during the commit process. This "
              + "indicates something is wrong that doesn't involve conflicts with other Iceberg operations. Enabling "
              + "{} may help in this case but the root cause should be investigated. Cleaning up {} groups which finished "
              + "being written.",
          PARTIAL_PROGRESS_ENABLED,
          PARTIAL_PROGRESS_ENABLED,

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Find the corresponding executor error in the Spark logs (the exception here is the wrapper).
2. Reduce group size via rewrite-options like max-file-group-size-bytes or lower partial-progress.max-commits so groups are smaller.
3. Increase executor memory/parallelism if failures are OOM-driven.
4. Re-run the action; only failed groups are rewritten again since successful groups already committed.

Example fix

null

// ensure table is readable and metadata current before rewrite
table.refresh();

try { result = action.execute(); } catch (Exception e) { /* inspect per-group logs; re-run action for failed groups only */ }

When it happens

Trigger: Running `rewrite_data_files` (or RewriteDataFilesSparkAction.execute) and one group's rewrite job throws — e.g. OOM in the Spark task, schema/partition eval errors, or write failure to the target location.

Common situations: Executor OOM on large data files, unsupported partition transforms in filters, transient object-store write errors, speculative-execution kill of a task.

Related errors

• Failure during rewrite process for group {}
• Cannot use column %s of type %s in ZOrdering, the type is un
• Cannot commit rewrite because of a ValidationException or Co
• org.apache.iceberg.spark.SparkRewriteTableCatalog does not s
• Table not found: ${ident}

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/7a5ae1d096aa4439. Report an issue: GitHub.