{"record":{"id":"27e43cccacf7fbf0","repo":"hashicorp/nomad","slug":"could-not-query-plugin-q-v","errorCode":null,"errorMessage":"could not query plugin %q: %v","messagePattern":"could not query plugin %q: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/csi_endpoint.go","lineNumber":1719,"sourceCode":"\t\treturn structs.ErrPermissionDenied\n\t}\n\n\tstateSnap, err := v.srv.fsm.State().Snapshot()\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tvar mErr multierror.Error\n\tfor _, snap := range args.Snapshots {\n\t\tif snap == nil {\n\t\t\t// we intentionally don't multierror here because we're in a weird state\n\t\t\treturn fmt.Errorf(\"snapshot cannot be nil\")\n\t\t}\n\n\t\tplugin, err := stateSnap.CSIPluginByID(nil, snap.PluginID)\n\t\tif err != nil {\n\t\t\tmultierror.Append(&mErr,\n\t\t\t\tfmt.Errorf(\"could not query plugin %q: %v\", snap.PluginID, err))\n\t\t\tcontinue\n\t\t}\n\t\tif plugin == nil {\n\t\t\tmultierror.Append(&mErr, fmt.Errorf(\"no such plugin\"))\n\t\t\tcontinue\n\t\t}\n\t\tif !plugin.HasControllerCapability(structs.CSIControllerSupportsCreateDeleteSnapshot) {\n\t\t\tmultierror.Append(&mErr, fmt.Errorf(\"plugin does not support snapshot\"))\n\t\t\tcontinue\n\t\t}\n\n\t\tmethod := \"ClientCSI.ControllerDeleteSnapshot\"\n\n\t\tcReq := &cstructs.ClientCSIControllerDeleteSnapshotRequest{\n\t\t\tID:      snap.ID,\n\t\t\tSecrets: snap.Secrets,\n\t\t}\n\t\tcReq.PluginID = plugin.ID","sourceCodeStart":1701,"sourceCodeEnd":1737,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/csi_endpoint.go#L1701-L1737","documentation":"Raised in CSIVolume.DeleteSnapshot when the state store lookup CSIPluginByID for the snapshot's PluginID returns an error (as opposed to a nil plugin). Nomad cannot even determine whether the CSI controller plugin that created the snapshot exists, so it records this per-snapshot error in the multierror and skips the delete for that snapshot.","triggerScenarios":"Calling CSIVolume.DeleteSnapshot (nomad volume snapshot delete API/CLI) with a snapshot whose PluginID fails the state store query — typically an internal state store corruption/error from Snapshot().CSIPluginByID, not a simple missing plugin (that yields 'no such plugin' instead).","commonSituations":"State store read failing under Raft/fstate store pressure; a stale snapshot record referencing a plugin during cluster state issues; snapshot records carried over after restores/upgrades where plugin queries misbehave.","solutions":["Inspect the wrapped inner error (%v) to identify the underlying state store failure and fix that root cause first.","Retry the DeleteSnapshot call once the state store is healthy; the error is per-snapshot and remaining snapshots still get processed.","Verify the snapshot's PluginID is correct and the plugin is registered (nomad plugin status); delete and re-register the plugin if its state entry is inconsistent.","As a last resort, remove the orphaned snapshot record from consideration or purge stale plugin state via server restart/recovery."],"exampleFix":"// before — deleting a snapshot whose plugin cannot be queried\nclient.CSI().DeleteSnapshot(req) // -> could not query plugin \"aws-ebs\": state store err\n// after — check plugin health before deleting\nplugin, _, _ := client.CSI().GetPlugin(\"aws-ebs\", nil)\nif plugin != nil { client.CSI().DeleteSnapshot(req) }","handlingStrategy":"validation","validationCode":"// verify the plugin state is queryable before deleting snapshots\nplug, _, err := client.CSI().GetPlugin(snap.PluginID, nil)\nif err != nil {\n    return fmt.Errorf(\"plugin %s not queryable: %w\", snap.PluginID, err)\n}\nif plug == nil { return fmt.Errorf(\"plugin %s not registered\", snap.PluginID) }","typeGuard":"func pluginQueryable(p *api.CSIPlugin, err error) bool { return err == nil && p != nil && p.ID != \"\" }","tryCatchPattern":null,"preventionTips":["Always check the wrapped inner error — this error is a symptom of a state store failure, not the real problem.","Prefer deleting snapshots while the originating plugin is registered and healthy.","Monitor Nomad server state store / Raft health.","Keep snapshot records and plugin lifecycles in sync; don't purge plugins with pending snapshot deletes."],"tags":["csi","state-store","snapshot","plugin"],"backgroundTag":"csi-plugin-unavailable","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}