apache/iceberg · warning

Removing invalid ref

Error message

Removing invalid ref {}: snapshot {} does not exist

What it means

During snapshot expiration, RemoveSnapshots.computeRetainedRefs resolves each named ref (branch/tag) to its snapshot. If the ref points to a snapshot ID that no longer exists in metadata, the ref is invalid and is dropped from the retained set with this warning — it will then be removed by the expiration. This normally happens only when metadata was externally modified or corrupted.

Solutions

1. Verify which refs are invalid: iterate table.refs() and check table.snapshot(ref.snapshotId()) != null; expect the expiration to remove them.
2. Restore the referenced snapshot by rolling back/re-applying metadata or restoring a metadata backup if the ref must survive.
3. Rebuild metadata consistency by re-registering the table from a valid metadata file version.
4. Avoid manual edits to table metadata; use Iceberg APIs (manageSnapshots, rollback) for ref management.

Example fix

// before: relying on a dangling ref
Snapshot snap = table.snapshot(ref.snapshotId()); // null -> warning, ref dropped
// after: clean the ref explicitly before expiring
if (table.snapshot(ref.snapshotId()) == null) {
  table.manageSnapshots().removeBranch(refName).commit(); // or removeTag
}

table.refs().forEach((name, ref) -> {
  if (table.snapshot(ref.snapshotId()) == null) {
    throw new IllegalStateException("Dangling ref " + name + " -> " + ref.snapshotId());
  }
});

boolean refIsValid(Table table, String name) {
  SnapshotRef ref = table.refs().get(name);
  return ref != null && table.snapshot(ref.snapshotId()) != null;
}

try { table.expireSnapshots().execute(); } catch (IllegalStateException e) { /* repair metadata before retry */ }

When it happens

Trigger: Calling expireSnapshots (or removeSnapshots) on a table whose refs reference snapshot IDs missing from snapshot-log — e.g. metadata edited out-of-band, a REST/catalog sync bug, or refs surviving aggressive metadata cleanup.

Common situations: Manual JSON metadata editing, third-party tools writing inconsistent metadata, restore/rollback flows interrupted midway, and cross-cluster table copies where only part of the metadata was copied.

Understand the failure class

Background: "Not found" and "does not exist" errors: why "Task not found", "No such folder", and "Can't find" fire when a lookup comes back empty — this error's family across 14 libraries.

Related errors

• Cannot create namespace " + namespace + ": metadata is not…
• Encountered invalid summary for snapshot
• Entries must have explicit snapshot ids if inherited…
• Failed to commit updates to table
• Failed to determine live files in manifest

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/a0f8818c3e3141bd. Report an issue: GitHub.

Appendix: source

Thrown at core/src/main/java/org/apache/iceberg/RemoveSnapshots.java:296

  private Map<String, SnapshotRef> computeRetainedRefs(Map<String, SnapshotRef> refs) {
    Map<String, SnapshotRef> retainedRefs = Maps.newHashMap();
    for (Map.Entry<String, SnapshotRef> refEntry : refs.entrySet()) {
      String name = refEntry.getKey();
      SnapshotRef ref = refEntry.getValue();
      if (name.equals(SnapshotRef.MAIN_BRANCH)) {
        retainedRefs.put(name, ref);
        continue;
      }

      Snapshot snapshot = base.snapshot(ref.snapshotId());
      long maxRefAgeMs = ref.maxRefAgeMs() != null ? ref.maxRefAgeMs() : defaultMaxRefAgeMs;
      if (snapshot != null) {
        long refAgeMs = now - snapshot.timestampMillis();
        if (refAgeMs <= maxRefAgeMs) {
          retainedRefs.put(name, ref);
        }
      } else {
        LOG.warn("Removing invalid ref {}: snapshot {} does not exist", name, ref.snapshotId());
      }
    }

    return retainedRefs;
  }

  private Set<Long> computeAllBranchSnapshotsToRetain(Collection<SnapshotRef> refs) {
    Set<Long> branchSnapshotsToRetain = Sets.newHashSet();
    for (SnapshotRef ref : refs) {
      if (ref.isBranch()) {
        long expireSnapshotsOlderThan =
            ref.maxSnapshotAgeMs() != null ? now - ref.maxSnapshotAgeMs() : defaultExpireOlderThan;
        int minSnapshotsToKeep =
            ref.minSnapshotsToKeep() != null ? ref.minSnapshotsToKeep() : defaultMinNumSnapshots;
        branchSnapshotsToRetain.addAll(
            computeBranchSnapshotsToRetain(
                ref.snapshotId(), expireSnapshotsOlderThan, minSnapshotsToKeep));
      }

View on GitHub (pinned to 86d9c8fc54)