apache/iceberg · warning

Skipping cleanup of written files

Error message

Skipping cleanup of written files

What it means

SparkPositionDeltaWrite.abort logs 'Skipping cleanup of written files' as a warning when the writer is configured NOT to delete uncommitted data files after a Spark job abort. Iceberg tracks orphan files written before commit; cleanup-on-abort is optional because deleting files whose commit state is uncertain can be unsafe. This message tells you the abort path deliberately left written files in place.

Source

Thrown at spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkPositionDeltaWrite.java:302

      }
    }

    private Expression conflictDetectionFilter(SparkBatchQueryScan queryScan) {
      Expression filter = Expressions.alwaysTrue();

      for (Expression expr : queryScan.filterExpressions()) {
        filter = Expressions.and(filter, expr);
      }

      return filter;
    }

    @Override
    public void abort(WriterCommitMessage[] messages) {
      if (cleanupOnAbort) {
        SparkCleanupUtil.deleteFiles("job abort", table.io(), files(messages));
      } else {
        LOG.warn("Skipping cleanup of written files");
      }
    }

    private List<ContentFile<?>> files(WriterCommitMessage[] messages) {
      List<ContentFile<?>> files = Lists.newArrayList();

      for (WriterCommitMessage message : messages) {
        if (message != null) {
          DeltaTaskCommit taskCommit = (DeltaTaskCommit) message;
          files.addAll(Arrays.asList(taskCommit.dataFiles()));
          files.addAll(Arrays.asList(taskCommit.deleteFiles()));
        }
      }

      return files;
    }

    private void commitOperation(SnapshotUpdate<?> operation, String description) {

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Enable cleanup on abort so aborted job files are deleted automatically (set the corresponding write.cleanup-on-abort style config / SparkSQLProperties cleanup flag to true).
2. Manually remove orphan files with RemoveOrphanFiles action: SparkActions.get(spark).deleteOrphanFiles(table).olderThan(ts).execute().
3. If leaving files is intentional, silence/ignore the warning; the files are ignored by Iceberg readers and can be GC'd later.
4. Verify job failure cause first — cleanup skipping is a secondary symptom; fix the aborting task.

Example fix

// before: cleanup skipped, orphan files accumulate
LOG.warn("Skipping cleanup of written files");

// after: enable cleanup via config before the write
spark.conf().set("spark.sql.iceberg.cleanup-on-abort.enabled", "true");

boolean cleanupEnabled = spark.conf().getOption("spark.sql.iceberg.cleanup-on-abort.enabled").getOrElse(() -> "false").equalsIgnoreCase("true");
if (!cleanupEnabled) {
  // schedule orphan-file GC for this table
  SparkActions.get(spark).deleteOrphanFiles(table).olderThan(Instant.now().minus(1, ChronoUnit.DAYS).toEpochMilli()).execute();
}

When it happens

Trigger: A Spark DELETE/MERGE/COPY-ON-WRITE position delta job fails or is killed after task writers have produced data files but before commit; the write.load.deletes-without-cleanup / cleanupOnAbort flag is false (default in some configurations), so abort() skips SparkCleanupUtil.deleteFiles.

Common situations: Jobs crashed mid-run leaving orphan files under table data/metadata dirs; users confused why storage grows after failed MERGE statements; running with cleanup disabled intentionally for crash-safety on object stores where deletion races with retries.

Related errors

• Skipping cleanup of written files
• Skipping cleanup of written files
• Skipping cleanup of written files
• Skipping cleanup of written files
• Cannot remove orphan files with an interval less than 24 hou

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/0287dfb2ee2c9654. Report an issue: GitHub.