apache/iceberg · warning

Failure during rewrite process for group {}

Error message

Failure during rewrite process for group {}

What it means

A logged warning emitted by RewriteDataFilesSparkAction when rewriting a file group fails. Tasks.foreach over plan.groups() with stopOnFailure/noRetry records the per-group exception in the onFailure handler; the group's partial result is discarded and remaining groups continue, so the compaction run degrades instead of failing wholesale.

Source

Thrown at spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java:280

    return new RewriteDataFilesCommitManager(
        table, startingSnapshotId, useStartingSequenceNumber, commitSummary(), branch);
  }

  private Builder doExecute(
      FileRewritePlan<FileGroupInfo, FileScanTask, DataFile, RewriteFileGroup> plan,
      RewriteDataFilesCommitManager commitManager) {
    ExecutorService rewriteService = rewriteService();

    ConcurrentLinkedQueue<RewriteFileGroup> rewrittenGroups = new ConcurrentLinkedQueue<>();

    Tasks.Builder<RewriteFileGroup> rewriteTaskBuilder =
        Tasks.foreach(plan.groups())
            .executeWith(rewriteService)
            .stopOnFailure()
            .noRetry()
            .onFailure(
                (fileGroup, exception) -> {
                  LOG.warn(
                      "Failure during rewrite process for group {}", fileGroup.info(), exception);
                });

    try {
      rewriteTaskBuilder.run(
          fileGroup -> {
            rewrittenGroups.add(rewriteFiles(plan, fileGroup));
          });
    } catch (Exception e) {
      // At least one rewrite group failed, clean up all completed rewrites
      LOG.error(
          "Cannot complete rewrite, {} is not enabled and one of the file set groups failed to "
              + "be rewritten. This error occurred during the writing of new files, not during the commit process. This "
              + "indicates something is wrong that doesn't involve conflicts with other Iceberg operations. Enabling "
              + "{} may help in this case but the root cause should be investigated. Cleaning up {} groups which finished "
              + "being written.",
          PARTIAL_PROGRESS_ENABLED,
          PARTIAL_PROGRESS_ENABLED,

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Read the chained exception for the specific group; fix the root cause (corrupt file, permissions, memory).
2. Reduce max-file-group-size / max-files-per-group to avoid executor OOM on large groups.
3. Re-run the rewrite; successfully completed groups are skipped, only failed groups are retried.
4. Skip or repair known-corrupt files before compaction (validate with Spark reads).
5. Increase executor memory if failures are OOM-driven: spark.executor.memory or memoryOverhead.

Example fix

// before
spark.table("db.tbl").call("rewrite_data_files");  // group fails on OOM
// after
spark.table("db.tbl").call("rewrite_data_files",
  Map.of("max-file-group-size-bytes", "1073741824"));  // 1GB groups
spark.conf.set("spark.executor.memory", "8g");

// validate data files are readable before compaction
spark.read.format("iceberg").load("db.tbl").limit(1000).collectAsList();

try { rewrite.execute(); } catch (Exception e) { /* group failures logged; fix cause and re-run */ }

When it happens

Trigger: Running optimize/rewriteDataFiles when one group's rewrite job throws — schema/eviction errors reading files, executor OOMs on large groups, corrupted data files, or write failures to the target location (permissions, quota).

Common situations: Corrupt legacy Parquet/Avro files from older writers; max-file-group-size too large causing executor OOM; target bucket/permission misconfigurations; incompatible codec or encryption settings; transient S3 throttling during writes.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Failure during rewrite process for group {}
• Rewrite data file error.
• Cannot commit rewrite because of a ValidationException or Co
• Cannot mix identity sort columns and a Zorder sort expressio
• unsupported strategy: + strategy + . Only binpack or sort is

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/18f5390041260426. Report an issue: GitHub.