{"record":{"id":"ee2edaf13a3fea9a","repo":"hashicorp/nomad","slug":"volume-lookup-failed-for-s-v","errorCode":null,"errorMessage":"volume lookup failed for %s: %v","messagePattern":"volume lookup failed for (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":2725,"sourceCode":"\titer, err := txn.Get(TableCSIVolumes, \"id\")\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"csi_volumes lookup failed: %v\", err)\n\t}\n\n\tws.Add(iter.WatchCh())\n\n\treturn iter, nil\n}\n\n// CSIVolumeByID is used to lookup a single volume. Returns a copy of the\n// volume because its plugins and allocations are denormalized to provide\n// accurate Health.\nfunc (s *StateStore) CSIVolumeByID(ws memdb.WatchSet, namespace, id string) (*structs.CSIVolume, error) {\n\ttxn := s.db.ReadTxn()\n\n\twatchCh, obj, err := txn.FirstWatch(TableCSIVolumes, \"id\", namespace, id)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"volume lookup failed for %s: %v\", id, err)\n\t}\n\tws.Add(watchCh)\n\n\tif obj == nil {\n\t\treturn nil, nil\n\t}\n\tvol := obj.(*structs.CSIVolume)\n\n\t// we return the volume with the plugins denormalized by default,\n\t// because the scheduler needs them for feasibility checking\n\treturn s.csiVolumeDenormalizePluginsTxn(txn, vol.Copy())\n}\n\n// CSIVolumesByPluginID looks up csi_volumes by pluginID. Caller should\n// snapshot if it wants to also denormalize the plugins.\nfunc (s *StateStore) CSIVolumesByPluginID(ws memdb.WatchSet, namespace, prefix, pluginID string) (memdb.ResultIterator, error) {\n\ttxn := s.db.ReadTxn()\n","sourceCodeStart":2707,"sourceCodeEnd":2743,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L2707-L2743","documentation":"Returned by StateStore.CSIVolumeByID when txn.FirstWatch on the csi_volumes \"id\" index fails. The %s is the volume ID and %v the memdb error. A missing volume does NOT trigger this — CSIVolumeByID returns (nil, nil) in that case — so seeing it always indicates a memdb-level read failure.","triggerScenarios":"CSIVolumeByID (backing volume read APIs) hits a FirstWatch error on TableCSIVolumes, i.e. invalid read transaction or schema/index mismatch.","commonSituations":"Forked/modified state schema; state corruption after a crash; incompatible snapshot restore.","solutions":["Check the wrapped %v for the underlying memdb reason","Restart the server so the state store is rebuilt from raft","Verify no local patches altered the csi_volumes table indexes","File a Nomad issue with nomad operator debug data if it persists"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"vol, _, err := api.CSIVolumesByID(ws, namespace, id)\nif err != nil && strings.Contains(err.Error(), \"volume lookup failed for\") {\n    // memdb read error, not absence: retry or restart server\n}","preventionTips":["Distinguish nil-result (volume absent) from this error (read failure)","Keep cluster versions aligned across servers","Avoid snapshot restores across incompatible versions","Capture nomad operator debug on recurrence"],"tags":["state-store","memdb","csi","read-txn","nomad"],"backgroundTag":"memdb-lookup-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}