{"record":{"id":"57f34f42f3f78872","repo":"hashicorp/nomad","slug":"error-querying-plugin-q-v","errorCode":null,"errorMessage":"error querying plugin %q: %v","messagePattern":"error querying plugin %q: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/csi_endpoint.go","lineNumber":1635,"sourceCode":"\t\tvol, err := state.CSIVolumeByID(nil, args.RequestNamespace(), snap.SourceVolumeID)\n\t\tif err != nil {\n\t\t\tmultierror.Append(&mErr, fmt.Errorf(\"error querying volume %q: %v\", snap.SourceVolumeID, err))\n\t\t\tcontinue\n\t\t}\n\t\tif vol == nil {\n\t\t\tmultierror.Append(&mErr, fmt.Errorf(\"no such volume %q\", snap.SourceVolumeID))\n\t\t\tcontinue\n\t\t}\n\n\t\tpluginID := snap.PluginID\n\t\tif pluginID == \"\" {\n\t\t\tpluginID = vol.PluginID\n\t\t}\n\n\t\tplugin, err := state.CSIPluginByID(nil, pluginID)\n\t\tif err != nil {\n\t\t\tmultierror.Append(&mErr,\n\t\t\t\tfmt.Errorf(\"error querying plugin %q: %v\", pluginID, err))\n\t\t\tcontinue\n\t\t}\n\t\tif plugin == nil {\n\t\t\tmultierror.Append(&mErr, fmt.Errorf(\"no such plugin %q\", pluginID))\n\t\t\tcontinue\n\t\t}\n\t\tif !plugin.HasControllerCapability(structs.CSIControllerSupportsCreateDeleteSnapshot) {\n\t\t\tmultierror.Append(&mErr,\n\t\t\t\tfmt.Errorf(\"plugin %q does not support snapshot\", pluginID))\n\t\t\tcontinue\n\t\t}\n\n\t\tsecrets := vol.Secrets\n\t\t// merge request secrets onto volume secrets\n\t\tmaps.Copy(secrets, snap.Secrets)\n\n\t\tcReq := &cstructs.ClientCSIControllerCreateSnapshotRequest{\n\t\t\tExternalSourceVolumeID: vol.ExternalID,","sourceCodeStart":1617,"sourceCodeEnd":1653,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/csi_endpoint.go#L1617-L1653","documentation":"CreateSnapshot validates each volume's CSI plugin by querying the state store via CSIPluginByID. When that lookup returns an error (state store failure), the error is appended to a multierror and the volume is skipped. This is not a plugin-missing error — it means the query itself failed, typically due to a snapshot of the state store or an internal store error.","triggerScenarios":"Calling the CreateSnapshot RPC (POST /v1/volumes/snapshot or nomad volume snapshot create) when the server's state store fails to read the CSIPlugin table, e.g. state store corruption or a boltdb/raft error during the read.","commonSituations":"Degraded Raft state on a server, state store file corruption, or read errors during leader transitions. Rare in practice; most users instead hit 'no such plugin'.","solutions":["Check server logs for the underlying state store error returned by CSIPluginByID to identify the real cause","Verify Raft cluster health (nomad server members, nomad operator raft list-peers) and that a stable leader exists","Restart/replace the failing Nomad server and restore from a known-good state snapshot if corruption is confirmed"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// before the snapshot RPC\npluginStatus, err := client.Plugins().Get(pluginID)\nif err != nil { return fmt.Errorf(\"plugin %q not queryable: %w\", pluginID, err) }","typeGuard":null,"tryCatchPattern":"// retry on transient state-store errors\nfor i := 0; i < 3; i++ {\n    _, err := client.Volumes().CreateSnapshot(req)\n    if err == nil { break }\n    if !isStateStoreError(err) { return err } // stop on 'no such plugin' etc.\n    time.Sleep(backoff(i))\n}","preventionTips":["Keep at least 3 servers with a stable Raft leader","Take periodic Nomad state snapshots for recovery","Check 'nomad operator raft list-peers' health before bulk snapshot operations"],"tags":["nomad","csi","state-store","raft"],"backgroundTag":"state-store-read-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}