apache/iceberg · warning

Failure during rewrite process for group {}

Error message

Failure during rewrite process for group {}

What it means

RewritePositionDeleteFilesSparkAction runs per-group rewrite tasks with stopOnFailure/noRetry and an onFailure callback that logs this warning with the group's info and exception. A group whose position-delete rewrite fails is reported here and simply omitted from the rewritten output rather than failing the job.

Source

Thrown at spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewritePositionDeleteFilesSparkAction.java:210

  private Result doExecute(
      FileRewritePlan<
              FileGroupInfo, PositionDeletesScanTask, DeleteFile, RewritePositionDeletesGroup>
          plan,
      RewritePositionDeletesCommitManager commitManager) {
    ExecutorService rewriteService = rewriteService();

    ConcurrentLinkedQueue<RewritePositionDeletesGroup> rewrittenGroups =
        new ConcurrentLinkedQueue<>();

    Tasks.Builder<RewritePositionDeletesGroup> rewriteTaskBuilder =
        Tasks.foreach(plan.groups())
            .executeWith(rewriteService)
            .stopOnFailure()
            .noRetry()
            .onFailure(
                (fileGroup, exception) ->
                    LOG.warn(
                        "Failure during rewrite process for group {}",
                        fileGroup.info(),
                        exception));

    try {
      rewriteTaskBuilder.run(fileGroup -> rewrittenGroups.add(rewriteDeleteFiles(plan, fileGroup)));
    } catch (Exception e) {
      // At least one rewrite group failed, clean up all completed rewrites
      LOG.error(
          "Cannot complete rewrite, {} is not enabled and one of the file set groups failed to "
              + "be rewritten. This error occurred during the writing of new files, not during the commit process. This "
              + "indicates something is wrong that doesn't involve conflicts with other Iceberg operations. Enabling "
              + "{} may help in this case but the root cause should be investigated. Cleaning up {} groups which finished "
              + "being written.",
          PARTIAL_PROGRESS_ENABLED,
          PARTIAL_PROGRESS_ENABLED,
          rewrittenGroups.size(),
          e);

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Read the attached exception and the group info (lists the delete files involved) to find the root cause.
2. Lower the group size threshold (rewrite-all / max file group size options) so each task handles less data.
3. Re-run the action after transient executor failures — failed groups are untouched and retried on the next run.
4. If delete files are corrupt, compact or remove the offending entries before re-running the rewrite.

When it happens

Trigger: A file group of position delete files fails during RewritePositionDeleteFiles doExecute — e.g. reader errors on corrupt deletion vectors, writer failures producing rewritten delete files, or executor OOM on large groups.

Common situations: Very large accumulated delete files exceeding executor memory; unreadable delete files from older writer versions; transient executor loss on shared clusters.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• Failure during rewrite process for group {}
• Cannot commit rewrite because of a ValidationException or Co
• Failure during rewrite process for group {}
• Failure during rewrite process for group {}
• Cannot commit rewrite because of a ValidationException or Co

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/778f96cb8d6f2d4e. Report an issue: GitHub.