{"record":{"id":"37a83de3fc74d12b","repo":"dagger/dagger","slug":"failed-to-remove-immutable-rec-with-missing-snapsh","errorCode":null,"errorMessage":"failed to remove immutable rec with missing snapshot","messagePattern":"failed to remove immutable rec with missing snapshot","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"engine/snapshots/manager.go","lineNumber":273,"sourceCode":"\t\ttriggerLastUsed: triggerUpdate,\n\t}\n\tbklog.G(context.TODO()).WithFields(ref.traceLogFields()).Trace(\"acquired cache ref\")\n\treturn ref, nil\n}\n\n// getRecord returns record for id. Requires manager lock.\nfunc (cm *snapshotManager) getRecord(ctx context.Context, id string, opts ...RefOption) (cr *cacheRecord, retErr error) {\n\tif rec, ok := cm.records[id]; ok {\n\t\tif rec.isDead() {\n\t\t\treturn nil, errors.Wrapf(errNotFound, \"failed to get dead record %s\", id)\n\t\t}\n\t\tif !rec.mutable {\n\t\t\tif _, err := cm.Snapshotter.Stat(ctx, rec.md.getSnapshotID()); err != nil {\n\t\t\t\tif !cerrdefs.IsNotFound(err) {\n\t\t\t\t\treturn nil, errors.Wrapf(err, \"failed to check immutable ref snapshot %s\", rec.md.getSnapshotID())\n\t\t\t\t}\n\t\t\t\tif err := rec.remove(ctx); err != nil {\n\t\t\t\t\treturn nil, errors.Wrap(err, \"failed to remove immutable rec with missing snapshot\")\n\t\t\t\t}\n\t\t\t\treturn nil, errors.Wrap(errNotFound, rec.md.getSnapshotID())\n\t\t\t}\n\t\t}\n\t\treturn rec, nil\n\t}\n\n\tmd, ok := cm.getMetadata(id)\n\tif !ok {\n\t\treturn nil, errors.Wrap(errNotFound, id)\n\t}\n\n\trec := &cacheRecord{\n\t\tmutable: !md.getCommitted(),\n\t\tcm:      cm,\n\t\tmd:      md,\n\t}\n","sourceCodeStart":255,"sourceCodeEnd":291,"githubUrl":"https://github.com/dagger/dagger/blob/82ba2681dbe30d3547a1dc50ea495900ab5b6047/engine/snapshots/manager.go#L255-L291","documentation":"When an immutable record's snapshot is confirmed missing (Stat returned NotFound), getRecord tries to remove the stale record from the cache metadata. If rec.remove itself fails, this error wraps the removal failure and aborts the lookup. It means the engine detected garbage but could not clean it up.","triggerScenarios":"get/GetMutable/GetMutableBySnapshotID on an in-memory immutable record whose underlying snapshot is gone AND rec.remove fails (metadata write failure, content store removal error, I/O error during cleanup).","commonSituations":"Read-only or full state directory preventing metadata deletion; content store blobs already partially deleted by a previous crashed prune; concurrent GC racing with the lookup.","solutions":["Inspect the wrapped rec.remove error; fix the underlying storage permission/disk condition","Stop the engine, check state/ metadata integrity, then restart so cache metadata is rebuilt","Run a full garbage collection / prune to remove orphaned records after disk issues are fixed","Restore the state directory from backup or wipe cache state if corruption is confirmed"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// ensure the state dir is writable before lookups that may self-heal\nif err := writableDir(stateDir); err != nil {\n\treturn fmt.Errorf(\"state dir not writable, cleanup would fail: %w\", err)\n}","typeGuard":"func isRecordRemoveFailure(err error) bool {\n\treturn err != nil && strings.Contains(err.Error(), \"failed to remove immutable rec with missing snapshot\")\n}","tryCatchPattern":"ref, err := cm.Get(ctx, id)\nif err != nil && isRecordRemoveFailure(err) {\n\tlog.Errorf(\"stale cache record cleanup failed: %v — check disk/permissions, then prune\", err)\n\treturn retryAfterMaintenance(ctx, id)\n}","preventionTips":["Keep the state directory writable and below disk-capacity thresholds","Do not run concurrent prunes during builds","After crashes, run a full prune before resuming heavy workloads"],"tags":["go","snapshotter","cache","cleanup"],"backgroundTag":"cache-record-cleanup-failed","analyzedSha":"82ba2681dbe30d3547a1dc50ea495900ab5b6047","analyzedAt":"2026-09-05T07:21:37.930Z","contentChangedAt":"2026-09-05T07:21:37.930Z","schemaVersion":2},"datasetVersion":"2026-09-12T12:17:11.808Z"}