dagger/dagger · error

failed to remove immutable rec with missing snapshot

Error message

failed to remove immutable rec with missing snapshot

What it means

When an immutable record's snapshot is confirmed missing (Stat returned NotFound), getRecord tries to remove the stale record from the cache metadata. If rec.remove itself fails, this error wraps the removal failure and aborts the lookup. It means the engine detected garbage but could not clean it up.

Source

Thrown at engine/snapshots/manager.go:273

		triggerLastUsed: triggerUpdate,
	}
	bklog.G(context.TODO()).WithFields(ref.traceLogFields()).Trace("acquired cache ref")
	return ref, nil
}

// getRecord returns record for id. Requires manager lock.
func (cm *snapshotManager) getRecord(ctx context.Context, id string, opts ...RefOption) (cr *cacheRecord, retErr error) {
	if rec, ok := cm.records[id]; ok {
		if rec.isDead() {
			return nil, errors.Wrapf(errNotFound, "failed to get dead record %s", id)
		}
		if !rec.mutable {
			if _, err := cm.Snapshotter.Stat(ctx, rec.md.getSnapshotID()); err != nil {
				if !cerrdefs.IsNotFound(err) {
					return nil, errors.Wrapf(err, "failed to check immutable ref snapshot %s", rec.md.getSnapshotID())
				}
				if err := rec.remove(ctx); err != nil {
					return nil, errors.Wrap(err, "failed to remove immutable rec with missing snapshot")
				}
				return nil, errors.Wrap(errNotFound, rec.md.getSnapshotID())
			}
		}
		return rec, nil
	}

	md, ok := cm.getMetadata(id)
	if !ok {
		return nil, errors.Wrap(errNotFound, id)
	}

	rec := &cacheRecord{
		mutable: !md.getCommitted(),
		cm:      cm,
		md:      md,
	}

View on GitHub (pinned to 82ba2681db)

Solutions

1. Inspect the wrapped rec.remove error; fix the underlying storage permission/disk condition
2. Stop the engine, check state/ metadata integrity, then restart so cache metadata is rebuilt
3. Run a full garbage collection / prune to remove orphaned records after disk issues are fixed
4. Restore the state directory from backup or wipe cache state if corruption is confirmed

// ensure the state dir is writable before lookups that may self-heal
if err := writableDir(stateDir); err != nil {
	return fmt.Errorf("state dir not writable, cleanup would fail: %w", err)
}

func isRecordRemoveFailure(err error) bool {
	return err != nil && strings.Contains(err.Error(), "failed to remove immutable rec with missing snapshot")
}

ref, err := cm.Get(ctx, id)
if err != nil && isRecordRemoveFailure(err) {
	log.Errorf("stale cache record cleanup failed: %v — check disk/permissions, then prune", err)
	return retryAfterMaintenance(ctx, id)
}

When it happens

Trigger: get/GetMutable/GetMutableBySnapshotID on an in-memory immutable record whose underlying snapshot is gone AND rec.remove fails (metadata write failure, content store removal error, I/O error during cleanup).

Common situations: Read-only or full state directory preventing metadata deletion; content store blobs already partially deleted by a previous crashed prune; concurrent GC racing with the lookup.

Related errors

• failed to remove mutable rec with missing snapshot
• %s release session %q: %w
• failed to check immutable ref snapshot %s
• failed to check mutable ref snapshot
• find stale dependency bindings: %w

AI-assisted analysis of dagger/dagger@82ba2681db (2026-09-05). Data as JSON: /api/errors/37a83de3fc74d12b. Report an issue: GitHub.