apache/iceberg · warning

Skipping cleanup of written files

Error message

Skipping cleanup of written files

What it means

This warning is logged by SparkWrite.abort when a Spark batch write job fails or is aborted and the 'cleanup-aborted-files' style flag (cleanupOnAbort) is disabled. Instead of deleting the data files written by the failed task writers, Iceberg leaves them in place and only warns, since these files are unreferenced by any snapshot. The write itself has failed regardless; this message only indicates orphan files may remain on disk.

Source

Thrown at spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java:271

      operation.toBranch(branch);
    }

    try {
      long start = System.currentTimeMillis();
      operation.commit(); // abort is automatically called if this fails
      long duration = System.currentTimeMillis() - start;
      LOG.info("Committed in {} ms", duration);
    } catch (Exception e) {
      cleanupOnAbort = e instanceof CleanableFailure;
      throw e;
    }
  }

  private void abort(WriterCommitMessage[] messages) {
    if (cleanupOnAbort) {
      SparkCleanupUtil.deleteFiles("job abort", table.io(), Lists.newArrayList(files(messages)));
    } else {
      LOG.warn("Skipping cleanup of written files");
    }
  }

  private DataFileSet files(WriterCommitMessage[] messages) {
    DataFileSet files = DataFileSet.create();

    for (WriterCommitMessage message : messages) {
      if (message != null) {
        TaskCommit taskCommit = (TaskCommit) message;
        files.addAll(Arrays.asList(taskCommit.files()));
      }
    }

    return files;
  }

  @Override
  public CustomTaskMetric[] reportDriverMetrics() {

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Enable cleanup on abort (set cleanupOnAbort true via the Spark SQL conf that controls cleanup of aborted writes) so failed writes delete their files
2. Run the DeleteOrphanFiles (remove_orphan_files) action periodically to garbage-collect the uncommitted files
3. Check driver logs for the root cause of the aborted write; fixing the write failure is the primary action

Example fix

// before (conf)
spark.sql.iceberg.cleanup-aborted-files=false
// after
spark.sql.iceberg.cleanup-aborted-files=true

boolean cleanup = spark.conf().get("spark.sql.iceberg.cleanup-aborted-files");
if (!cleanup) System.out.println("Aborted writes may leave orphan files; schedule remove_orphan_files");

When it happens

Trigger: A Spark write using SparkWrite's batch/CTAS path fails during commit or the query is cancelled, triggering Spark's abort() callback, while cleanupOnAbort (spark.sql.iceberg.cleanup-aborted-files / job-level config) is false.

Common situations: Users on Spark versions where the cleanup config defaults to off; clusters where users kill long-running writes; CTAS/RTAS failures leaving orphan parquet files that later need ExpireSnapshots/DeleteOrphanFiles to remove.

Related errors

• Skipping cleanup of written files
• Skipping cleanup of written files
• Cannot remove orphan files with an interval less than 24 hou
• Cannot remove orphan files with an interval less than 24 hou
• Cannot remove orphan files with an interval less than 24 hou

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/4542cd44f3c64294. Report an issue: GitHub.