apache/iceberg · warning

Failure during rewrite process for group {}

Error message

Failure during rewrite process for group {}

What it means

RewriteDataFilesSparkAction executes rewrite tasks per file group with stopOnFailure and no retry; before running, the onFailure callback logs this warning with the group's info and exception. A group whose rewrite task failed is reported here and excluded from the result instead of failing the whole Spark job.

Source

Thrown at spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java:273

    return new RewriteDataFilesCommitManager(
        table, startingSnapshotId, useStartingSequenceNumber, commitSummary(), branch);
  }

  private Builder doExecute(
      FileRewritePlan<FileGroupInfo, FileScanTask, DataFile, RewriteFileGroup> plan,
      RewriteDataFilesCommitManager commitManager) {
    ExecutorService rewriteService = rewriteService();

    ConcurrentLinkedQueue<RewriteFileGroup> rewrittenGroups = new ConcurrentLinkedQueue<>();

    Tasks.Builder<RewriteFileGroup> rewriteTaskBuilder =
        Tasks.foreach(plan.groups())
            .executeWith(rewriteService)
            .stopOnFailure()
            .noRetry()
            .onFailure(
                (fileGroup, exception) -> {
                  LOG.warn(
                      "Failure during rewrite process for group {}", fileGroup.info(), exception);
                });

    try {
      rewriteTaskBuilder.run(
          fileGroup -> {
            rewrittenGroups.add(rewriteFiles(plan, fileGroup));
          });
    } catch (Exception e) {
      // At least one rewrite group failed, clean up all completed rewrites
      LOG.error(
          "Cannot complete rewrite, {} is not enabled and one of the file set groups failed to "
              + "be rewritten. This error occurred during the writing of new files, not during the commit process. This "
              + "indicates something is wrong that doesn't involve conflicts with other Iceberg operations. Enabling "
              + "{} may help in this case but the root cause should be investigated. Cleaning up {} groups which finished "
              + "being written.",
          PARTIAL_PROGRESS_ENABLED,
          PARTIAL_PROGRESS_ENABLED,

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Read the attached exception to identify the failing group and root cause (the group info lists its files).
2. Reduce max-file-group-size / partial-progress.max-commits so groups are smaller and less memory-hungry.
3. Set rewrite options like partial-progress.enabled true so successful groups still commit despite this group failing.
4. Fix corrupt source files or exclude them (delete entries) then re-run the rewrite.

Example fix

// before: one huge group fails and aborts rewrite work
spark.sql("CALL catalog.system.rewrite_data_files(table => 'db.t')");
// after: enable partial progress and smaller groups
spark.sql("CALL catalog.system.rewrite_data_files(table => 'db.t', " +
  "options => map('partial-progress.enabled','true','max-file-group-size-bytes','1073741824'))");

spark.sql("CALL cat.sys.rewrite_data_files(table => 'db.t', options => map('partial-progress.enabled','true','max-file-group-size-bytes','1073741824'))");

When it happens

Trigger: A file-group rewrite task throws during RewriteDataFiles doExecute — e.g. OOM while rewriting large groups, schema/read errors on corrupt data files, or writer failures writing new parquet files.

Common situations: Groups with files too large for executor memory (max-file-group-size too high); corrupt or non-readable files written by older writers; executor lost / speculative failures in a busy cluster.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• Failure during rewrite process for group {}
• Cannot commit rewrite because of a ValidationException or Co
• %s is true but %d rewrite commits failed. This is more than
• Cannot mix identity sort columns and a Zorder sort expressio
• unsupported strategy: ${strategy}. Only binpack or sort is s

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/aa783379348e2195. Report an issue: GitHub.