{"record":{"id":"44dd2ff8457c5de5","repo":"apache/iceberg","slug":"skipping-cleanup-of-written-files-44dd2f","errorCode":null,"errorMessage":"Skipping cleanup of written files","messagePattern":"Skipping cleanup of written files","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java","lineNumber":252,"sourceCode":"      operation.toBranch(branch);\n    }\n\n    try {\n      long start = System.currentTimeMillis();\n      operation.commit(); // abort is automatically called if this fails\n      long duration = System.currentTimeMillis() - start;\n      LOG.info(\"Committed in {} ms\", duration);\n    } catch (Exception e) {\n      cleanupOnAbort = e instanceof CleanableFailure;\n      throw e;\n    }\n  }\n\n  private void abort(WriterCommitMessage[] messages) {\n    if (cleanupOnAbort) {\n      SparkCleanupUtil.deleteFiles(\"job abort\", table.io(), Lists.newArrayList(files(messages)));\n    } else {\n      LOG.warn(\"Skipping cleanup of written files\");\n    }\n  }\n\n  private DataFileSet files(WriterCommitMessage[] messages) {\n    DataFileSet files = DataFileSet.create();\n\n    for (WriterCommitMessage message : messages) {\n      if (message != null) {\n        TaskCommit taskCommit = (TaskCommit) message;\n        files.addAll(Arrays.asList(taskCommit.files()));\n      }\n    }\n\n    return files;\n  }\n\n  @Override\n  public String toString() {","sourceCodeStart":234,"sourceCodeEnd":270,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java#L234-L270","documentation":"SparkWrite.abort logs 'Skipping cleanup of written files' when a Spark job using the Iceberg write aborts and cleanupOnAbort is disabled. Data files written by tasks before the failed commit remain in storage as orphans; Iceberg readers ignore them because no snapshot references them.","triggerScenarios":"A Spark append/overwrite/CTAS job fails or is cancelled after writers commit task files but the job commit fails; SparkWrite.abort runs with cleanupOnAbort == false, so SparkCleanupUtil.deleteFiles is not called.","commonSituations":"Long-running inserts killed by OOM or preemption; concurrent-commit conflicts causing job-level abort; object-store tables where operators disabled cleanup to avoid deleting files another concurrent job might reference.","solutions":["Enable cleanup-on-abort (spark.sql.iceberg.cleanup-on-abort.enabled or the write option) so abort deletes task files.","Run RemoveOrphanFiles to clean existing orphans: SparkActions.get(spark).deleteOrphanFiles(table).olderThan(...).execute().","Tune max concurrency/retry settings to reduce commit conflicts that trigger abort.","If intentional, monitor storage growth and schedule periodic orphan-file GC."],"exampleFix":"// before\nspark.conf().set(\"spark.sql.iceberg.cleanup-on-abort.enabled\", \"false\");\n\n// after\nspark.conf().set(\"spark.sql.iceberg.cleanup-on-abort.enabled\", \"true\");","handlingStrategy":"validation","validationCode":"boolean cleanupEnabled = Boolean.parseBoolean(spark.conf().get(\"spark.sql.iceberg.cleanup-on-abort.enabled\", \"false\"));\nif (!cleanupEnabled) {\n  System.out.println(\"Aborted writes will leave orphan files; schedule RemoveOrphanFiles\");\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Set spark.sql.iceberg.cleanup-on-abort.enabled=true when safe","Schedule periodic RemoveOrphanFiles for the warehouse","Reduce commit conflicts (retries, lower concurrent writers)","Alert on the warn message in job logs"],"tags":["spark","orphan-files","cleanup","write-abort"],"backgroundTag":"orphan-file-cleanup-skipped","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}