hashicorp/nomad · error

could not query plugin %q: %v

Error message

could not query plugin %q: %v

What it means

Raised in CSIVolume.DeleteSnapshot when the state store lookup CSIPluginByID for the snapshot's PluginID returns an error (as opposed to a nil plugin). Nomad cannot even determine whether the CSI controller plugin that created the snapshot exists, so it records this per-snapshot error in the multierror and skips the delete for that snapshot.

Source

Thrown at nomad/csi_endpoint.go:1719

		return structs.ErrPermissionDenied
	}

	stateSnap, err := v.srv.fsm.State().Snapshot()
	if err != nil {
		return err
	}

	var mErr multierror.Error
	for _, snap := range args.Snapshots {
		if snap == nil {
			// we intentionally don't multierror here because we're in a weird state
			return fmt.Errorf("snapshot cannot be nil")
		}

		plugin, err := stateSnap.CSIPluginByID(nil, snap.PluginID)
		if err != nil {
			multierror.Append(&mErr,
				fmt.Errorf("could not query plugin %q: %v", snap.PluginID, err))
			continue
		}
		if plugin == nil {
			multierror.Append(&mErr, fmt.Errorf("no such plugin"))
			continue
		}
		if !plugin.HasControllerCapability(structs.CSIControllerSupportsCreateDeleteSnapshot) {
			multierror.Append(&mErr, fmt.Errorf("plugin does not support snapshot"))
			continue
		}

		method := "ClientCSI.ControllerDeleteSnapshot"

		cReq := &cstructs.ClientCSIControllerDeleteSnapshotRequest{
			ID:      snap.ID,
			Secrets: snap.Secrets,
		}
		cReq.PluginID = plugin.ID

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped inner error (%v) to identify the underlying state store failure and fix that root cause first.
2. Retry the DeleteSnapshot call once the state store is healthy; the error is per-snapshot and remaining snapshots still get processed.
3. Verify the snapshot's PluginID is correct and the plugin is registered (nomad plugin status); delete and re-register the plugin if its state entry is inconsistent.
4. As a last resort, remove the orphaned snapshot record from consideration or purge stale plugin state via server restart/recovery.

Example fix

// before — deleting a snapshot whose plugin cannot be queried
client.CSI().DeleteSnapshot(req) // -> could not query plugin "aws-ebs": state store err
// after — check plugin health before deleting
plugin, _, _ := client.CSI().GetPlugin("aws-ebs", nil)
if plugin != nil { client.CSI().DeleteSnapshot(req) }

// verify the plugin state is queryable before deleting snapshots
plug, _, err := client.CSI().GetPlugin(snap.PluginID, nil)
if err != nil {
    return fmt.Errorf("plugin %s not queryable: %w", snap.PluginID, err)
}
if plug == nil { return fmt.Errorf("plugin %s not registered", snap.PluginID) }

func pluginQueryable(p *api.CSIPlugin, err error) bool { return err == nil && p != nil && p.ID != "" }

When it happens

Trigger: Calling CSIVolume.DeleteSnapshot (nomad volume snapshot delete API/CLI) with a snapshot whose PluginID fails the state store query — typically an internal state store corruption/error from Snapshot().CSIPluginByID, not a simple missing plugin (that yields 'no such plugin' instead).

Common situations: State store read failing under Raft/fstate store pressure; a stale snapshot record referencing a plugin during cluster state issues; snapshot records carried over after restores/upgrades where plugin queries misbehave.

Related errors

• CSI.ControllerCreateSnapshot: %w: %v
• CSI.ControllerCreateSnapshot: plugin did not return error or
• CSI.ControllerDeleteSnapshot: %w: %v
• CSI.ControllerListSnapshots: %w: %v
• CSI.ControllerListSnapshot: plugin returned an invalid entry

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/27e43cccacf7fbf0. Report an issue: GitHub.