{"record":{"id":"7be41673cb3baa25","repo":"hashicorp/nomad","slug":"alloc-lookup-failed-v","errorCode":null,"errorMessage":"alloc lookup failed: %v","messagePattern":"alloc lookup failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":2811,"sourceCode":"\n\t// Filter the iterator by ID prefix\n\tf := func(raw any) bool {\n\t\tv, ok := raw.(*structs.CSIVolume)\n\t\tif !ok {\n\t\t\treturn false\n\t\t}\n\t\treturn !strings.HasPrefix(v.ID, prefix)\n\t}\n\twrap := memdb.NewFilterIterator(iter, f)\n\treturn wrap, nil\n}\n\n// CSIVolumesByNodeID looks up CSIVolumes in use on a node. Caller should\n// snapshot if it wants to also denormalize the plugins.\nfunc (s *StateStore) CSIVolumesByNodeID(ws memdb.WatchSet, prefix, nodeID string) (memdb.ResultIterator, error) {\n\tallocs, err := s.AllocsByNode(ws, nodeID)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"alloc lookup failed: %v\", err)\n\t}\n\n\t// Find volume ids for CSI volumes in running allocs, or allocs that we desire to run\n\tids := map[string]string{} // Map volumeID to Namespace\n\tfor _, a := range allocs {\n\t\ttg := a.Job.LookupTaskGroup(a.TaskGroup)\n\n\t\tif !(a.DesiredStatus == structs.AllocDesiredStatusRun ||\n\t\t\ta.ClientStatus == structs.AllocClientStatusRunning) ||\n\t\t\tlen(tg.Volumes) == 0 {\n\t\t\tcontinue\n\t\t}\n\n\t\tfor _, v := range tg.Volumes {\n\t\t\tif v.Type != structs.VolumeTypeCSI {\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\tids[v.Source] = a.Namespace","sourceCodeStart":2793,"sourceCodeEnd":2829,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L2793-L2829","documentation":"Returned by StateStore.CSIVolumesByNodeID when the delegated call to s.AllocsByNode(ws, nodeID) fails while collecting allocations for a node to discover which CSI volumes are in use. The underlying alloc lookup error is preserved in %v.","triggerScenarios":"Listing CSI volumes by node ID when AllocsByNode's memdb Get on the allocs table (\"node\" index) returns an error.","commonSituations":"Corrupt allocs index after a bad upgrade or forked schema; OOM-induced memdb failure; usually seen together with other state-store read errors.","solutions":["Check the wrapped %v for the root memdb error from the allocs table","Restart the server to rebuild state from raft","Verify consistent Nomad versions across all server nodes","Capture a debug bundle and escalate to Nomad if it recurs"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"vols, _, err := api.CSIVolumesByNodeID(prefix, nodeID)\nif err != nil && strings.Contains(err.Error(), \"alloc lookup failed\") {\n    // allocs-table read failure: backoff-retry, then restart server if persistent\n    return retryWithBackoff(func() error { _, _, e := api.CSIVolumesByNodeID(prefix, nodeID); return e })\n}","preventionTips":["Confirm node is registered (nomad node status) before querying its volumes","Watch for correlated allocs-table errors as an early corruption signal","Maintain even Nomad versions across servers","Restart and rebuild from raft on persistent failures"],"tags":["state-store","memdb","allocs","csi","nomad"],"backgroundTag":"memdb-lookup-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}