{"record":{"id":"24ec3a3ba0aefc9c","repo":"hashicorp/nomad","slug":"error-querying-volume-q-v","errorCode":null,"errorMessage":"error querying volume %q: %v","messagePattern":"error querying volume %q: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/csi_endpoint.go","lineNumber":1619,"sourceCode":"\t\treturn structs.ErrPermissionDenied\n\t}\n\n\tstate, err := v.srv.fsm.State().Snapshot()\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tmethod := \"ClientCSI.ControllerCreateSnapshot\"\n\tvar mErr multierror.Error\n\tfor _, snap := range args.Snapshots {\n\t\tif snap == nil {\n\t\t\t// we intentionally don't multierror here because we're in a weird state\n\t\t\treturn fmt.Errorf(\"snapshot cannot be nil\")\n\t\t}\n\n\t\tvol, err := state.CSIVolumeByID(nil, args.RequestNamespace(), snap.SourceVolumeID)\n\t\tif err != nil {\n\t\t\tmultierror.Append(&mErr, fmt.Errorf(\"error querying volume %q: %v\", snap.SourceVolumeID, err))\n\t\t\tcontinue\n\t\t}\n\t\tif vol == nil {\n\t\t\tmultierror.Append(&mErr, fmt.Errorf(\"no such volume %q\", snap.SourceVolumeID))\n\t\t\tcontinue\n\t\t}\n\n\t\tpluginID := snap.PluginID\n\t\tif pluginID == \"\" {\n\t\t\tpluginID = vol.PluginID\n\t\t}\n\n\t\tplugin, err := state.CSIPluginByID(nil, pluginID)\n\t\tif err != nil {\n\t\t\tmultierror.Append(&mErr,\n\t\t\t\tfmt.Errorf(\"error querying plugin %q: %v\", pluginID, err))\n\t\t\tcontinue\n\t\t}","sourceCodeStart":1601,"sourceCodeEnd":1637,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/csi_endpoint.go#L1601-L1637","documentation":"CreateSnapshot fails to look up the source volume in Nomad's state store via CSIVolumeByID and appends 'error querying volume %q: %v' to the multierror, continuing with the remaining snapshots. The inner error is a state-store query failure (RPC/connection error to the state backend or a store-level failure), not merely 'volume absent' — an absent volume takes the separate 'no such volume' path (vol == nil).","triggerScenarios":"Sending CSISnapshotCreateRequest with Snapshots whose SourceVolumeID causes CSIVolumeByID(nil, ns, id) to return a non-nil error: state store backend errors, ACL-driven or namespace query failures surfaced as errors by the store, or a server-internal RPC error while reading volumes.","commonSituations":"Nomad raft/state store issues (server instability, corrupted state) during snapshot creation; querying with a namespace the store errors on rather than returning empty; transient server-side errors during multi-snapshot batch requests where later entries continue to be processed.","solutions":["Read the wrapped %v inner error in the multierror to identify the actual state-store failure and fix that root cause (server health, raft state, logs around the CSIVolumeByID call)","Retry CreateSnapshot once the server/state store is healthy — state query errors are frequently transient","Verify the request's RequestNamespace() is valid and reachable for the caller's ACL token; adjust the token's namespace rules","Collect and inspect all entries of the returned multierror.Error — some snapshots may have succeeded/failed independently"],"exampleFix":"// before\nmErr := nomadCreateSnapshot(...) // opaque handling of multierror\n// after\nif err != nil { for _, e := range err.(*multierror.Error).Errors { log.Printf(\"snapshot error: %v\", e) } } // then fix the underlying state-store cause per entry","handlingStrategy":"retry","validationCode":"vol, _, err := client.Volumes().Get(ctx, snap.SourceVolumeID, nil)\nif err != nil { return fmt.Errorf(\"cannot snapshot: source volume %q not queryable: %w\", snap.SourceVolumeID, err) }","typeGuard":null,"tryCatchPattern":"err := createSnapshot(req)\nif me, ok := err.(*multierror.Error); ok {\n    for _, e := range me.Errors {\n        if strings.Contains(e.Error(), \"error querying volume\") {\n            // inspect wrapped cause; retry transient state-store failures with backoff\n        }\n    }\n}","preventionTips":["Pre-check every SourceVolumeID with a volume Get before issuing the batch","Parse the full multierror, not just the first message","Monitor Nomad server/raft health; state query errors often cluster with server instability"],"tags":["nomad","csi","snapshot","state-store","multierror"],"backgroundTag":"state-store-query-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}