{"record":{"id":"6eb78fa5d79723bf","repo":"apache/iceberg","slug":"skipping-cleanup-of-written-files-6eb78f","errorCode":null,"errorMessage":"Skipping cleanup of written files","messagePattern":"Skipping cleanup of written files","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkPositionDeltaWrite.java","lineNumber":323,"sourceCode":"        commitOperation(rowDelta, commitMsg, summary);\n\n      } else {\n        String commitMsg =\n            String.format(\n                Locale.ROOT,\n                \"position delta with %d data files and %d delete files (no validation required)\",\n                addedDataFilesCount,\n                addedDeleteFilesCount);\n        commitOperation(rowDelta, commitMsg, summary);\n      }\n    }\n\n    @Override\n    public void abort(WriterCommitMessage[] messages) {\n      if (cleanupOnAbort) {\n        SparkCleanupUtil.deleteFiles(\"job abort\", table.io(), files(messages));\n      } else {\n        LOG.warn(\"Skipping cleanup of written files\");\n      }\n    }\n\n    private List<ContentFile<?>> files(WriterCommitMessage[] messages) {\n      List<ContentFile<?>> files = Lists.newArrayList();\n\n      for (WriterCommitMessage message : messages) {\n        if (message != null) {\n          DeltaTaskCommit taskCommit = (DeltaTaskCommit) message;\n          files.addAll(Arrays.asList(taskCommit.dataFiles()));\n          files.addAll(Arrays.asList(taskCommit.deleteFiles()));\n        }\n      }\n\n      return files;\n    }\n\n    private void commitOperation(","sourceCodeStart":305,"sourceCodeEnd":341,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkPositionDeltaWrite.java#L305-L341","documentation":"A WARN logged in SparkPositionDeltaWrite.abort when the write is aborted by a non-CleanableFailure exception, so cleanupOnAbort is false and the task-written files are deliberately NOT deleted. This is intentional: Iceberg cannot always distinguish a retriable/transactional failure from one where files may be needed for the commit, so files are left for later orphan cleanup.","triggerScenarios":"Spark's write abort(WriterCommitMessage[]) is invoked and the triggering exception is not an instanceof CleanableFailure, making cleanupOnAbort false.","commonSituations":"Job failures from non-Iceberg exceptions (OOM, executor loss, user code errors) during MERGE/DELETE position-delta writes; files accumulate in the table location until orphan-file expiration runs.","solutions":["Confirm whether the aborting exception should implement CleanableFailure","Schedule/execute removeOrphanFiles to delete abandoned files","Set the job up so failures are CleanableFailure (retryable transactional errors) when cleanup is desired"],"exampleFix":null,"handlingStrategy":"validation","validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":["Schedule removeOrphanFiles with an appropriate olderThan interval","Ensure expected/retriable failures implement CleanableFailure so aborts clean up","Monitor table locations for accumulating orphan files"],"tags":["spark","write-abort","cleanup"],"backgroundTag":"orphan-file-cleanup","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}