What causes "Failure during rewrite process for group {}" in apache/iceberg?

Inspect the Spark executor logs for the underlying task exception.

Failure during rewrite process for group {}

Error message

Failure during rewrite process for group {}

What it means

WARN log from RewritePositionDeleteFilesSparkAction.doExecute: a group of position-delete files failed during its rewrite Spark job; the Tasks.foreach onFailure hook logs the group's info and exception. Like the data-file rewrite, failures are surfaced per-group without necessarily failing the whole statement.

Source

Thrown at spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/RewritePositionDeleteFilesSparkAction.java:210

  private Result doExecute(
      FileRewritePlan<
              FileGroupInfo, PositionDeletesScanTask, DeleteFile, RewritePositionDeletesGroup>
          plan,
      RewritePositionDeletesCommitManager commitManager) {
    ExecutorService rewriteService = rewriteService();

    ConcurrentLinkedQueue<RewritePositionDeletesGroup> rewrittenGroups =
        new ConcurrentLinkedQueue<>();

    Tasks.Builder<RewritePositionDeletesGroup> rewriteTaskBuilder =
        Tasks.foreach(plan.groups())
            .executeWith(rewriteService)
            .stopOnFailure()
            .noRetry()
            .onFailure(
                (fileGroup, exception) ->
                    LOG.warn(
                        "Failure during rewrite process for group {}",
                        fileGroup.info(),
                        exception));

    try {
      rewriteTaskBuilder.run(fileGroup -> rewrittenGroups.add(rewriteDeleteFiles(plan, fileGroup)));
    } catch (Exception e) {
      // At least one rewrite group failed, clean up all completed rewrites
      LOG.error(
          "Cannot complete rewrite, {} is not enabled and one of the file set groups failed to "
              + "be rewritten. This error occurred during the writing of new files, not during the commit process. This "
              + "indicates something is wrong that doesn't involve conflicts with other Iceberg operations. Enabling "
              + "{} may help in this case but the root cause should be investigated. Cleaning up {} groups which finished "
              + "being written.",
          PARTIAL_PROGRESS_ENABLED,
          PARTIAL_PROGRESS_ENABLED,
          rewrittenGroups.size(),
          e);

View on GitHub (pinned to 86d9c8fc54)

Solutions

Inspect the Spark executor logs for the underlying task exception.
Rewrite fewer files per group / increase executor memory if OOM.
Verify the failing position-delete file is readable; exclude or repair corrupt files.
Re-run the action; successfully rewritten groups are already committed.

Example fix

null

Defensive patterns

Strategy: retry

Validate before calling

table.refresh(); // current snapshot includes expected delete files

Try / catch

try { result = action.execute(); } catch (Exception e) { /* per-group failures are logged; re-run for failed groups */ }

Prevention

Size position-delete groups to fit executor memory
Validate delete files readable after writer upgrades
Re-run to complete groups that failed

When it happens

Trigger: Running rewrite_position_delete_files where one group's job throws — compaction of position deletes fails due to OOM, reader errors on corrupt delete files, or write failures.

Common situations: Very large position-delete files exhausting executor memory; delete files written by older writers with incompatible layouts; transient storage errors during the rewritten-parquet write.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/b30fb5b72414f6d8. Report an issue: GitHub.