apache/iceberg · warning

Skipping cleanup of written files

Error message

Skipping cleanup of written files

What it means

A WARN logged in SparkWrite.abort when the write aborts with a non-CleanableFailure exception, so cleanupOnAbort is false and files written by tasks are intentionally left in place rather than deleted. Iceberg treats non-cleanable failures conservatively to avoid deleting files that might still be needed.

Source

Thrown at spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java:282

      operation.toBranch(branch);
    }

    try {
      long start = System.currentTimeMillis();
      operation.commit(); // abort is automatically called if this fails
      long duration = System.currentTimeMillis() - start;
      LOG.info("Committed in {} ms", duration);
    } catch (Exception e) {
      cleanupOnAbort = e instanceof CleanableFailure;
      throw e;
    }
  }

  private void abort(WriterCommitMessage[] messages) {
    if (cleanupOnAbort) {
      SparkCleanupUtil.deleteFiles("job abort", table.io(), Lists.newArrayList(files(messages)));
    } else {
      LOG.warn("Skipping cleanup of written files");
    }
  }

  private DataFileSet files(WriterCommitMessage[] messages) {
    DataFileSet files = DataFileSet.create();

    for (WriterCommitMessage message : messages) {
      if (message != null) {
        TaskCommit taskCommit = (TaskCommit) message;
        files.addAll(Arrays.asList(taskCommit.files()));
      }
    }

    return files;
  }

  @Override
  public CustomTaskMetric[] reportDriverMetrics() {

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Execute removeOrphanFiles to purge abandoned files
2. Make expected failure modes CleanableFailure so aborts clean up automatically
3. Tune orphan-file expiration interval to bound storage leakage

When it happens

Trigger: abort(WriterCommitMessage[]) runs and cleanupOnAbort == false, i.e. the exception that caused the abort does not implement CleanableFailure.

Common situations: Executor loss, OOM, or user-code exceptions during batch append writes; orphaned data files accumulate until removeOrphanFiles expiration.

Related errors

• Skipping cleanup of written files
• Skipping cleanup of written files
• Skipping cleanup of written files
• Failed to delete {} ({})
• Deleted only {} of {} file(s) ({})

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/99810f8c9ad20178. Report an issue: GitHub.