{"record":{"id":"f3665b803c3849c2","repo":"hashicorp/nomad","slug":"volume-lookup-failed-s-v-f3665b","errorCode":null,"errorMessage":"volume lookup failed: %s: %v","messagePattern":"volume lookup failed: (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":2876,"sourceCode":"\n\titer, err := txn.Get(TableCSIVolumes, \"id_prefix\", namespace, prefix)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"volume lookup failed: %v\", err)\n\t}\n\n\tws.Add(iter.WatchCh())\n\n\treturn iter, nil\n}\n\n// CSIVolumeClaim updates the volume's claim count and allocation list\nfunc (s *StateStore) CSIVolumeClaim(index uint64, now int64, namespace, id string, claim *structs.CSIVolumeClaim) error {\n\ttxn := s.db.WriteTxnMsgT(structs.CSIVolumeClaimRequestType, index)\n\tdefer txn.Abort()\n\n\trow, err := txn.First(TableCSIVolumes, \"id\", namespace, id)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"volume lookup failed: %s: %v\", id, err)\n\t}\n\tif row == nil {\n\t\treturn fmt.Errorf(\"volume not found: %s\", id)\n\t}\n\n\torig, ok := row.(*structs.CSIVolume)\n\tif !ok {\n\t\treturn fmt.Errorf(\"volume row conversion error\")\n\t}\n\n\tvar alloc *structs.Allocation\n\tif claim.State == structs.CSIVolumeClaimStateTaken {\n\t\talloc, err = s.allocByIDImpl(txn, nil, claim.AllocationID)\n\t\tif err != nil {\n\t\t\ts.logger.Error(\"AllocByID failed\", \"error\", err)\n\t\t\treturn fmt.Errorf(structs.ErrUnknownAllocationPrefix)\n\t\t}\n\t\tif alloc == nil {","sourceCodeStart":2858,"sourceCodeEnd":2894,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L2858-L2894","documentation":"Returned by StateStore.CSIVolumeClaim when txn.First(TableCSIVolumes, \"id\", namespace, id) errors while locating the volume to attach/detach a claim (used by the CSI volume claim RPC during alloc setup/teardown). The %s is the volume ID and %v the memdb error. A nil row is handled separately (error 2599), so this error is strictly a read failure.","triggerScenarios":"Client submits CSIVolumeClaimRequest (claim/unclaim on alloc run or stop) and the First lookup on the csi_volumes id index fails.","commonSituations":"State store corruption after crash/restore; fork-modified schema; OOM pressure on the leader.","solutions":["Inspect the wrapped %v message in the server logs","Restart the server to rebuild state from the raft log and retry the claim","Re-run the affected allocation (nomad alloc stop / redeploy) once the store is healthy","Escalate to Nomad with a debug bundle if reproducible"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// check the volume exists before submitting a claim\nvol, _, err := api.CSIVolumesByID(ws, namespace, volumeID)\nif err != nil {\n    return fmt.Errorf(\"state store read error, retry later: %w\", err)\n}\nif vol == nil {\n    return fmt.Errorf(\"volume %s does not exist in namespace %s\", volumeID, namespace)\n}","typeGuard":null,"tryCatchPattern":"err := api.CSIVolumeClaim(&structs.CSIVolumeClaimRequest{VolumeID: id, ...})\nif err != nil && strings.Contains(err.Error(), \"volume lookup failed\") {\n    // memdb read failure: backoff-retry, then restart server\n    return retryWithBackoff(func() error { return api.CSIVolumeClaim(req) })\n}","preventionTips":["Verify volume existence before claiming","Retry idempotent claim RPCs with backoff","Ensure server state is healthy before scheduling CSI workloads","Match namespace exactly when claiming"],"tags":["state-store","memdb","csi","claim","nomad"],"backgroundTag":"memdb-lookup-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}