{"record":{"id":"4542cd44f3c64294","repo":"apache/iceberg","slug":"skipping-cleanup-of-written-files-4542cd","errorCode":null,"errorMessage":"Skipping cleanup of written files","messagePattern":"Skipping cleanup of written files","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java","lineNumber":271,"sourceCode":"      operation.toBranch(branch);\n    }\n\n    try {\n      long start = System.currentTimeMillis();\n      operation.commit(); // abort is automatically called if this fails\n      long duration = System.currentTimeMillis() - start;\n      LOG.info(\"Committed in {} ms\", duration);\n    } catch (Exception e) {\n      cleanupOnAbort = e instanceof CleanableFailure;\n      throw e;\n    }\n  }\n\n  private void abort(WriterCommitMessage[] messages) {\n    if (cleanupOnAbort) {\n      SparkCleanupUtil.deleteFiles(\"job abort\", table.io(), Lists.newArrayList(files(messages)));\n    } else {\n      LOG.warn(\"Skipping cleanup of written files\");\n    }\n  }\n\n  private DataFileSet files(WriterCommitMessage[] messages) {\n    DataFileSet files = DataFileSet.create();\n\n    for (WriterCommitMessage message : messages) {\n      if (message != null) {\n        TaskCommit taskCommit = (TaskCommit) message;\n        files.addAll(Arrays.asList(taskCommit.files()));\n      }\n    }\n\n    return files;\n  }\n\n  @Override\n  public CustomTaskMetric[] reportDriverMetrics() {","sourceCodeStart":253,"sourceCodeEnd":289,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java#L253-L289","documentation":"This warning is logged by SparkWrite.abort when a Spark batch write job fails or is aborted and the 'cleanup-aborted-files' style flag (cleanupOnAbort) is disabled. Instead of deleting the data files written by the failed task writers, Iceberg leaves them in place and only warns, since these files are unreferenced by any snapshot. The write itself has failed regardless; this message only indicates orphan files may remain on disk.","triggerScenarios":"A Spark write using SparkWrite's batch/CTAS path fails during commit or the query is cancelled, triggering Spark's abort() callback, while cleanupOnAbort (spark.sql.iceberg.cleanup-aborted-files / job-level config) is false.","commonSituations":"Users on Spark versions where the cleanup config defaults to off; clusters where users kill long-running writes; CTAS/RTAS failures leaving orphan parquet files that later need ExpireSnapshots/DeleteOrphanFiles to remove.","solutions":["Enable cleanup on abort (set cleanupOnAbort true via the Spark SQL conf that controls cleanup of aborted writes) so failed writes delete their files","Run the DeleteOrphanFiles (remove_orphan_files) action periodically to garbage-collect the uncommitted files","Check driver logs for the root cause of the aborted write; fixing the write failure is the primary action"],"exampleFix":"// before (conf)\nspark.sql.iceberg.cleanup-aborted-files=false\n// after\nspark.sql.iceberg.cleanup-aborted-files=true","handlingStrategy":"validation","validationCode":"boolean cleanup = spark.conf().get(\"spark.sql.iceberg.cleanup-aborted-files\");\nif (!cleanup) System.out.println(\"Aborted writes may leave orphan files; schedule remove_orphan_files\");","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Enable cleanup-aborted-files in shared Spark configs","Schedule regular DeleteOrphanFiles maintenance","Monitor driver logs for 'Skipping cleanup of written files'"],"tags":["spark","file-cleanup","write-abort","orphan-files"],"backgroundTag":"unsupported-config-value","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}