apache/iceberg · warning

Skipping cleanup of written files

Error message

Skipping cleanup of written files

What it means

SparkWrite.abort logs 'Skipping cleanup of written files' when a Spark job using the Iceberg write aborts and cleanupOnAbort is disabled. Data files written by tasks before the failed commit remain in storage as orphans; Iceberg readers ignore them because no snapshot references them.

Source

Thrown at spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java:252

      operation.toBranch(branch);
    }

    try {
      long start = System.currentTimeMillis();
      operation.commit(); // abort is automatically called if this fails
      long duration = System.currentTimeMillis() - start;
      LOG.info("Committed in {} ms", duration);
    } catch (Exception e) {
      cleanupOnAbort = e instanceof CleanableFailure;
      throw e;
    }
  }

  private void abort(WriterCommitMessage[] messages) {
    if (cleanupOnAbort) {
      SparkCleanupUtil.deleteFiles("job abort", table.io(), Lists.newArrayList(files(messages)));
    } else {
      LOG.warn("Skipping cleanup of written files");
    }
  }

  private DataFileSet files(WriterCommitMessage[] messages) {
    DataFileSet files = DataFileSet.create();

    for (WriterCommitMessage message : messages) {
      if (message != null) {
        TaskCommit taskCommit = (TaskCommit) message;
        files.addAll(Arrays.asList(taskCommit.files()));
      }
    }

    return files;
  }

  @Override
  public String toString() {

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Enable cleanup-on-abort (spark.sql.iceberg.cleanup-on-abort.enabled or the write option) so abort deletes task files.
2. Run RemoveOrphanFiles to clean existing orphans: SparkActions.get(spark).deleteOrphanFiles(table).olderThan(...).execute().
3. Tune max concurrency/retry settings to reduce commit conflicts that trigger abort.
4. If intentional, monitor storage growth and schedule periodic orphan-file GC.

Example fix

// before
spark.conf().set("spark.sql.iceberg.cleanup-on-abort.enabled", "false");

// after
spark.conf().set("spark.sql.iceberg.cleanup-on-abort.enabled", "true");

boolean cleanupEnabled = Boolean.parseBoolean(spark.conf().get("spark.sql.iceberg.cleanup-on-abort.enabled", "false"));
if (!cleanupEnabled) {
  System.out.println("Aborted writes will leave orphan files; schedule RemoveOrphanFiles");
}

When it happens

Trigger: A Spark append/overwrite/CTAS job fails or is cancelled after writers commit task files but the job commit fails; SparkWrite.abort runs with cleanupOnAbort == false, so SparkCleanupUtil.deleteFiles is not called.

Common situations: Long-running inserts killed by OOM or preemption; concurrent-commit conflicts causing job-level abort; object-store tables where operators disabled cleanup to avoid deleting files another concurrent job might reference.

Related errors

• Skipping cleanup of written files
• Skipping cleanup of written files
• Skipping cleanup of written files
• Skipping cleanup of written files
• Cannot remove orphan files with an interval less than 24 hou

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/44dd2ff8457c5de5. Report an issue: GitHub.