{"record":{"id":"74886ce683e2b684","repo":"apache/iceberg","slug":"failed-to-load-committed-metadata-skipping-clean","errorCode":null,"errorMessage":"Failed to load committed metadata, skipping clean-up","messagePattern":"Failed to load committed metadata, skipping clean-up","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"core/src/main/java/org/apache/iceberg/BaseTransaction.java","lineNumber":411,"sourceCode":"      // A manifest could be deleted in one successful operation commit, but reused in another\n      // successful commit of that operation if the whole transaction is retried.\n      Set<Snapshot> newSnapshots = Sets.newHashSet();\n      for (Snapshot snapshot : current.snapshots()) {\n        if (!startingSnapshots.contains(snapshot.snapshotId())) {\n          newSnapshots.add(snapshot);\n        }\n      }\n\n      Set<String> committedFiles = committedFiles(ops.io(), newSnapshots);\n      // delete all the files that were deleted in the most recent set of operation commits\n      Set<String> uncommittedFiles =\n          deletedFiles.stream()\n              .filter(f -> !committedFiles.contains(f))\n              .collect(Collectors.toSet());\n      deleteUncommittedFiles(uncommittedFiles);\n\n    } catch (RuntimeException e) {\n      LOG.warn(\"Failed to load committed metadata, skipping clean-up\", e);\n    }\n  }\n\n  protected void cleanUp() {\n    // the commit failed and no files were committed. clean up each update.\n    cleanAllUpdates();\n\n    // delete all the uncommitted files\n    deleteUncommittedFiles(deletedFiles);\n  }\n\n  private void cleanAllUpdates() {\n    Tasks.foreach(updates)\n        .suppressFailureWhenFinished()\n        .run(\n            update -> {\n              if (update instanceof SnapshotProducer) {\n                ((SnapshotProducer) update).cleanAll();","sourceCodeStart":393,"sourceCodeEnd":429,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/core/src/main/java/org/apache/iceberg/BaseTransaction.java#L393-L429","documentation":"A WARN log in commitSimpleTransaction: after the transaction commit itself, the code tries to reload the committed table metadata to determine which data files were actually written/committed so orphaned files can be deleted. If reloading the committed metadata fails (any RuntimeException), Iceberg cannot compute the committed set, so it skips deleting uncommitted files and logs this warning — potentially leaving orphan files behind rather than risking data loss.","triggerScenarios":"A transaction commit where the post-commit metadata reload (refresh from the new metadata location) throws — e.g. transient I/O errors reading the metadata file, permission errors, or a metastore returning an inconsistent state right after commit.","commonSituations":"S3 eventual-consistency or permission issues when re-reading the just-written metadata JSON; concurrent commit moved the location; network blips between the commit and the cleanup step.","solutions":["Check for and manually delete orphan data files using Iceberg's orphan file removal procedures (e.g. remove_orphan_files in Spark).","Retry the metadata load manually; if the table loads fine afterward, only the cleanup was skipped.","Verify FileIO permissions on the metadata location.","Enable metrics/audit to detect accumulating uncommitted files if this recurs frequently."],"exampleFix":"// before: assuming commit fully cleans up\ntransaction.commitTransaction();\n// after: periodic orphan cleanup\ntable.refresh();\n// run: DELETE unreferenced files\nSparkActions.get().deleteOrphanFiles(table).olderThan(System.currentTimeMillis() - 86400_000).execute();","handlingStrategy":"fallback","validationCode":"// after commit, verify metadata is loadable\nTable reloaded = catalog.loadTable(identifier); // throws if cleanup metadata unreadable","typeGuard":null,"tryCatchPattern":"try { transaction.commitTransaction(); } catch (RuntimeException e) { // schedule orphan-file cleanup instead of retrying blindly }","preventionTips":["Schedule periodic deleteOrphanFiles runs on all tables","Ensure FileIO has read access to metadata location","Watch for repeated occurrences (indicates persistent IO problems)","Keep metadata storage strongly consistent (S3 with proper consistency settings)"],"tags":["cleanup","orphan-files","commit"],"backgroundTag":"file-read-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}