{"record":{"id":"99810f8c9ad20178","repo":"apache/iceberg","slug":"skipping-cleanup-of-written-files-99810f","errorCode":null,"errorMessage":"Skipping cleanup of written files","messagePattern":"Skipping cleanup of written files","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java","lineNumber":282,"sourceCode":"      operation.toBranch(branch);\n    }\n\n    try {\n      long start = System.currentTimeMillis();\n      operation.commit(); // abort is automatically called if this fails\n      long duration = System.currentTimeMillis() - start;\n      LOG.info(\"Committed in {} ms\", duration);\n    } catch (Exception e) {\n      cleanupOnAbort = e instanceof CleanableFailure;\n      throw e;\n    }\n  }\n\n  private void abort(WriterCommitMessage[] messages) {\n    if (cleanupOnAbort) {\n      SparkCleanupUtil.deleteFiles(\"job abort\", table.io(), Lists.newArrayList(files(messages)));\n    } else {\n      LOG.warn(\"Skipping cleanup of written files\");\n    }\n  }\n\n  private DataFileSet files(WriterCommitMessage[] messages) {\n    DataFileSet files = DataFileSet.create();\n\n    for (WriterCommitMessage message : messages) {\n      if (message != null) {\n        TaskCommit taskCommit = (TaskCommit) message;\n        files.addAll(Arrays.asList(taskCommit.files()));\n      }\n    }\n\n    return files;\n  }\n\n  @Override\n  public CustomTaskMetric[] reportDriverMetrics() {","sourceCodeStart":264,"sourceCodeEnd":300,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java#L264-L300","documentation":"A WARN logged in SparkWrite.abort when the write aborts with a non-CleanableFailure exception, so cleanupOnAbort is false and files written by tasks are intentionally left in place rather than deleted. Iceberg treats non-cleanable failures conservatively to avoid deleting files that might still be needed.","triggerScenarios":"abort(WriterCommitMessage[]) runs and cleanupOnAbort == false, i.e. the exception that caused the abort does not implement CleanableFailure.","commonSituations":"Executor loss, OOM, or user-code exceptions during batch append writes; orphaned data files accumulate until removeOrphanFiles expiration.","solutions":["Execute removeOrphanFiles to purge abandoned files","Make expected failure modes CleanableFailure so aborts clean up automatically","Tune orphan-file expiration interval to bound storage leakage"],"exampleFix":null,"handlingStrategy":"validation","validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":["Run removeOrphanFiles periodically with a safe olderThan window","Prefer failure modes that implement CleanableFailure","Alert on this warning to catch storage leakage early"],"tags":["spark","write-abort","cleanup"],"backgroundTag":"orphan-file-cleanup","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}