apache/iceberg · warning

Failed to load committed metadata, skipping clean-up

Error message

Failed to load committed metadata, skipping clean-up

What it means

A WARN log in commitSimpleTransaction: after the transaction commit itself, the code tries to reload the committed table metadata to determine which data files were actually written/committed so orphaned files can be deleted. If reloading the committed metadata fails (any RuntimeException), Iceberg cannot compute the committed set, so it skips deleting uncommitted files and logs this warning — potentially leaving orphan files behind rather than risking data loss.

Source

Thrown at core/src/main/java/org/apache/iceberg/BaseTransaction.java:411

      // A manifest could be deleted in one successful operation commit, but reused in another
      // successful commit of that operation if the whole transaction is retried.
      Set<Snapshot> newSnapshots = Sets.newHashSet();
      for (Snapshot snapshot : current.snapshots()) {
        if (!startingSnapshots.contains(snapshot.snapshotId())) {
          newSnapshots.add(snapshot);
        }
      }

      Set<String> committedFiles = committedFiles(ops.io(), newSnapshots);
      // delete all the files that were deleted in the most recent set of operation commits
      Set<String> uncommittedFiles =
          deletedFiles.stream()
              .filter(f -> !committedFiles.contains(f))
              .collect(Collectors.toSet());
      deleteUncommittedFiles(uncommittedFiles);

    } catch (RuntimeException e) {
      LOG.warn("Failed to load committed metadata, skipping clean-up", e);
    }
  }

  protected void cleanUp() {
    // the commit failed and no files were committed. clean up each update.
    cleanAllUpdates();

    // delete all the uncommitted files
    deleteUncommittedFiles(deletedFiles);
  }

  private void cleanAllUpdates() {
    Tasks.foreach(updates)
        .suppressFailureWhenFinished()
        .run(
            update -> {
              if (update instanceof SnapshotProducer) {
                ((SnapshotProducer) update).cleanAll();

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Check for and manually delete orphan data files using Iceberg's orphan file removal procedures (e.g. remove_orphan_files in Spark).
2. Retry the metadata load manually; if the table loads fine afterward, only the cleanup was skipped.
3. Verify FileIO permissions on the metadata location.
4. Enable metrics/audit to detect accumulating uncommitted files if this recurs frequently.

Example fix

// before: assuming commit fully cleans up
transaction.commitTransaction();
// after: periodic orphan cleanup
table.refresh();
// run: DELETE unreferenced files
SparkActions.get().deleteOrphanFiles(table).olderThan(System.currentTimeMillis() - 86400_000).execute();

// after commit, verify metadata is loadable
Table reloaded = catalog.loadTable(identifier); // throws if cleanup metadata unreadable

try { transaction.commitTransaction(); } catch (RuntimeException e) { // schedule orphan-file cleanup instead of retrying blindly }

When it happens

Trigger: A transaction commit where the post-commit metadata reload (refresh from the new metadata location) throws — e.g. transient I/O errors reading the metadata file, permission errors, or a metastore returning an inconsistent state right after commit.

Common situations: S3 eventual-consistency or permission issues when re-reading the just-written metadata JSON; concurrent commit moved the location; network blips between the commit and the cleanup step.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Failed to load committed snapshot, skipping manifest clean-u
• Failed to load committed table metadata or during cleanup, s
• Failed to commit rewrite, cleaning up rewritten files
• Skipping cleanup of written files
• Skipping cleanup of written files

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/74886ce683e2b684. Report an issue: GitHub.