apache/iceberg · warning

Skipping cleanup of written files

Error message

Skipping cleanup of written files

What it means

A WARN logged in SparkPositionDeltaWrite.abort when the write is aborted by a non-CleanableFailure exception, so cleanupOnAbort is false and the task-written files are deliberately NOT deleted. This is intentional: Iceberg cannot always distinguish a retriable/transactional failure from one where files may be needed for the commit, so files are left for later orphan cleanup.

Source

Thrown at spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkPositionDeltaWrite.java:323

        commitOperation(rowDelta, commitMsg, summary);

      } else {
        String commitMsg =
            String.format(
                Locale.ROOT,
                "position delta with %d data files and %d delete files (no validation required)",
                addedDataFilesCount,
                addedDeleteFilesCount);
        commitOperation(rowDelta, commitMsg, summary);
      }
    }

    @Override
    public void abort(WriterCommitMessage[] messages) {
      if (cleanupOnAbort) {
        SparkCleanupUtil.deleteFiles("job abort", table.io(), files(messages));
      } else {
        LOG.warn("Skipping cleanup of written files");
      }
    }

    private List<ContentFile<?>> files(WriterCommitMessage[] messages) {
      List<ContentFile<?>> files = Lists.newArrayList();

      for (WriterCommitMessage message : messages) {
        if (message != null) {
          DeltaTaskCommit taskCommit = (DeltaTaskCommit) message;
          files.addAll(Arrays.asList(taskCommit.dataFiles()));
          files.addAll(Arrays.asList(taskCommit.deleteFiles()));
        }
      }

      return files;
    }

    private void commitOperation(

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Confirm whether the aborting exception should implement CleanableFailure
2. Schedule/execute removeOrphanFiles to delete abandoned files
3. Set the job up so failures are CleanableFailure (retryable transactional errors) when cleanup is desired

When it happens

Trigger: Spark's write abort(WriterCommitMessage[]) is invoked and the triggering exception is not an instanceof CleanableFailure, making cleanupOnAbort false.

Common situations: Job failures from non-Iceberg exceptions (OOM, executor loss, user code errors) during MERGE/DELETE position-delta writes; files accumulate in the table location until orphan-file expiration runs.

Related errors

• Skipping cleanup of written files
• Skipping cleanup of written files
• Skipping cleanup of written files
• Failed to delete {} ({})
• Deleted only {} of {} file(s) ({})

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/6eb78fa5d79723bf. Report an issue: GitHub.