apache/iceberg · warning

Failure during rewrite process for group {}

Error message

Failure during rewrite process for group {}

What it means

RewriteDataFilesSparkAction.doExecute schedules one rewrite task per file group; the Tasks builder stops on the first failure and logs 'Failure during rewrite process for group <group>' with the exception. The affected group's files are not rewritten and the original files remain committed-untouched, so data is safe but compaction is incomplete.

Source

Thrown at spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java:280

    return new RewriteDataFilesCommitManager(
        table, startingSnapshotId, useStartingSequenceNumber, commitSummary(), branch);
  }

  private Builder doExecute(
      FileRewritePlan<FileGroupInfo, FileScanTask, DataFile, RewriteFileGroup> plan,
      RewriteDataFilesCommitManager commitManager) {
    ExecutorService rewriteService = rewriteService();

    ConcurrentLinkedQueue<RewriteFileGroup> rewrittenGroups = new ConcurrentLinkedQueue<>();

    Tasks.Builder<RewriteFileGroup> rewriteTaskBuilder =
        Tasks.foreach(plan.groups())
            .executeWith(rewriteService)
            .stopOnFailure()
            .noRetry()
            .onFailure(
                (fileGroup, exception) -> {
                  LOG.warn(
                      "Failure during rewrite process for group {}", fileGroup.info(), exception);
                });

    try {
      rewriteTaskBuilder.run(
          fileGroup -> {
            rewrittenGroups.add(rewriteFiles(plan, fileGroup));
          });
    } catch (Exception e) {
      // At least one rewrite group failed, clean up all completed rewrites
      LOG.error(
          "Cannot complete rewrite, {} is not enabled and one of the file set groups failed to "
              + "be rewritten. This error occurred during the writing of new files, not during the commit process. This "
              + "indicates something is wrong that doesn't involve conflicts with other Iceberg operations. Enabling "
              + "{} may help in this case but the root cause should be investigated. Cleaning up {} groups which finished "
              + "being written.",
          PARTIAL_PROGRESS_ENABLED,
          PARTIAL_PROGRESS_ENABLED,

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Read the chained exception for the failing group and fix the root cause (usually memory or file corruption)
2. Reduce max-file-group-size-bytes / partial-progress settings so groups are smaller and failures are isolated
3. Re-run the rewrite action after fixing — unmodified groups are simply retried

Example fix

// before
actions.rewriteDataFiles(table).execute();
// after
actions.rewriteDataFiles(table)
    .option("max-file-group-size-bytes", "1073741824")
    .option("partial-progress.enabled", "true")
    .execute();

// keep groups small enough for executor memory
long groupBytes = /* max-file-group-size-bytes */ 0;
if (groupBytes > executorMemory * 2) { /* lower the option */ }

try {
  RewriteDataFilesSparkAction rw = actions.rewriteDataFiles(table);
  rw.option("partial-progress.enabled", "true").execute();
} catch (Exception e) {
  // failing groups logged; fix cause and re-run remaining groups
}

When it happens

Trigger: A rewrite group's task throws during doExecute — e.g. writer failures from schema issues, oversized groups exceeding memory, or read errors on the input files.

Common situations: Very large file groups causing OOM in executors; corrupted or concurrently-modified input files; partial-failure caching/AVRO schema evolution mismatches during rewrites.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Failure during rewrite process for group {}
• Failure during rewrite process for group {}
• Failure during rewrite process for group {}
• Failure during rewrite process for group {}
• Cannot commit rewrite because of a ValidationException or Co

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/a8b7c949e555bb5d. Report an issue: GitHub.