{"record":{"id":"dbf8ea39243116af","repo":"hashicorp/nomad","slug":"volume-lookup-failed-s-v","errorCode":null,"errorMessage":"volume lookup failed: %s %v","messagePattern":"volume lookup failed: (.+?) (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":2840,"sourceCode":"\t\t\tcontinue\n\t\t}\n\n\t\tfor _, v := range tg.Volumes {\n\t\t\tif v.Type != structs.VolumeTypeCSI {\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\tids[v.Source] = a.Namespace\n\t\t}\n\t}\n\n\t// Lookup the raw CSIVolumes to match the other list interfaces\n\titer := NewSliceIterator()\n\ttxn := s.db.ReadTxn()\n\tfor id, namespace := range ids {\n\t\tif strings.HasPrefix(id, prefix) {\n\t\t\twatchCh, raw, err := txn.FirstWatch(TableCSIVolumes, \"id\", namespace, id)\n\t\t\tif err != nil {\n\t\t\t\treturn nil, fmt.Errorf(\"volume lookup failed: %s %v\", id, err)\n\t\t\t}\n\t\t\tws.Add(watchCh)\n\t\t\titer.Add(raw)\n\t\t}\n\t}\n\n\treturn iter, nil\n}\n\n// CSIVolumesByNamespace looks up the entire csi_volumes table\nfunc (s *StateStore) CSIVolumesByNamespace(ws memdb.WatchSet, namespace, prefix string) (memdb.ResultIterator, error) {\n\ttxn := s.db.ReadTxn()\n\n\treturn s.csiVolumesByNamespaceImpl(txn, ws, namespace, prefix)\n}\n\nfunc (s *StateStore) csiVolumesByNamespaceImpl(txn *txn, ws memdb.WatchSet, namespace, prefix string) (memdb.ResultIterator, error) {\n","sourceCodeStart":2822,"sourceCodeEnd":2858,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L2822-L2858","documentation":"Returned by StateStore.CSIVolumesByNodeID while iterating the discovered volume IDs: for each candidate ID a txn.FirstWatch(TableCSIVolumes, \"id\", namespace, id) is issued, and any failure aborts the whole listing with this message naming the volume ID. The %s identifies which specific volume lookup failed.","triggerScenarios":"An alloc on the node references a volume whose per-ID FirstWatch read fails inside memdb — invalid read transaction or index/schema mismatch mid-iteration.","commonSituations":"State store corruption after crash/upgrade; a fork changed the csi_volumes id index; concurrent memdb failure under memory pressure.","solutions":["Note the volume ID in the message and inspect the wrapped %v for the memdb cause","Restart the server to rebuild state from the raft log","Check whether any referenced volumes have stale/invalid records after an upgrade and re-register them if needed","File a Nomad issue with the debug bundle if persistent"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"vols, _, err := api.CSIVolumesByNodeID(prefix, nodeID)\nif err != nil {\n    var volID string\n    if _, e := fmt.Sscanf(err.Error(), \"volume lookup failed: %s\", &volID); e == nil {\n        log.Printf(\"failed reading volume %s: re-register it\", volID)\n    }\n    return err\n}","preventionTips":["Re-register volumes referenced by running allocs after upgrades","Purge allocations referencing deregistered volumes","Check the named volume ID in the message first when debugging","Keep schema-modifying patches out of production builds"],"tags":["state-store","memdb","csi","node","nomad"],"backgroundTag":"memdb-lookup-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}