{"record":{"id":"0287dfb2ee2c9654","repo":"apache/iceberg","slug":"skipping-cleanup-of-written-files","errorCode":null,"errorMessage":"Skipping cleanup of written files","messagePattern":"Skipping cleanup of written files","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkPositionDeltaWrite.java","lineNumber":302,"sourceCode":"      }\n    }\n\n    private Expression conflictDetectionFilter(SparkBatchQueryScan queryScan) {\n      Expression filter = Expressions.alwaysTrue();\n\n      for (Expression expr : queryScan.filterExpressions()) {\n        filter = Expressions.and(filter, expr);\n      }\n\n      return filter;\n    }\n\n    @Override\n    public void abort(WriterCommitMessage[] messages) {\n      if (cleanupOnAbort) {\n        SparkCleanupUtil.deleteFiles(\"job abort\", table.io(), files(messages));\n      } else {\n        LOG.warn(\"Skipping cleanup of written files\");\n      }\n    }\n\n    private List<ContentFile<?>> files(WriterCommitMessage[] messages) {\n      List<ContentFile<?>> files = Lists.newArrayList();\n\n      for (WriterCommitMessage message : messages) {\n        if (message != null) {\n          DeltaTaskCommit taskCommit = (DeltaTaskCommit) message;\n          files.addAll(Arrays.asList(taskCommit.dataFiles()));\n          files.addAll(Arrays.asList(taskCommit.deleteFiles()));\n        }\n      }\n\n      return files;\n    }\n\n    private void commitOperation(SnapshotUpdate<?> operation, String description) {","sourceCodeStart":284,"sourceCodeEnd":320,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkPositionDeltaWrite.java#L284-L320","documentation":"SparkPositionDeltaWrite.abort logs 'Skipping cleanup of written files' as a warning when the writer is configured NOT to delete uncommitted data files after a Spark job abort. Iceberg tracks orphan files written before commit; cleanup-on-abort is optional because deleting files whose commit state is uncertain can be unsafe. This message tells you the abort path deliberately left written files in place.","triggerScenarios":"A Spark DELETE/MERGE/COPY-ON-WRITE position delta job fails or is killed after task writers have produced data files but before commit; the write.load.deletes-without-cleanup / cleanupOnAbort flag is false (default in some configurations), so abort() skips SparkCleanupUtil.deleteFiles.","commonSituations":"Jobs crashed mid-run leaving orphan files under table data/metadata dirs; users confused why storage grows after failed MERGE statements; running with cleanup disabled intentionally for crash-safety on object stores where deletion races with retries.","solutions":["Enable cleanup on abort so aborted job files are deleted automatically (set the corresponding write.cleanup-on-abort style config / SparkSQLProperties cleanup flag to true).","Manually remove orphan files with RemoveOrphanFiles action: SparkActions.get(spark).deleteOrphanFiles(table).olderThan(ts).execute().","If leaving files is intentional, silence/ignore the warning; the files are ignored by Iceberg readers and can be GC'd later.","Verify job failure cause first — cleanup skipping is a secondary symptom; fix the aborting task."],"exampleFix":"// before: cleanup skipped, orphan files accumulate\nLOG.warn(\"Skipping cleanup of written files\");\n\n// after: enable cleanup via config before the write\nspark.conf().set(\"spark.sql.iceberg.cleanup-on-abort.enabled\", \"true\");","handlingStrategy":"validation","validationCode":"boolean cleanupEnabled = spark.conf().getOption(\"spark.sql.iceberg.cleanup-on-abort.enabled\").getOrElse(() -> \"false\").equalsIgnoreCase(\"true\");\nif (!cleanupEnabled) {\n  // schedule orphan-file GC for this table\n  SparkActions.get(spark).deleteOrphanFiles(table).olderThan(Instant.now().minus(1, ChronoUnit.DAYS).toEpochMilli()).execute();\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Enable cleanup-on-abort for tables on storage where orphan deletion is safe","Run RemoveOrphanFiles on a schedule (e.g. olderThan 3 days)","Monitor table storage growth for uncommitted files","Fix root causes of job aborts (OOM, conflicts) rather than only cleaning up"],"tags":["spark","orphan-files","cleanup","write-abort"],"backgroundTag":"orphan-file-cleanup-skipped","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}