hashicorp/nomad · error

error querying plugin %q: %v

Error message

error querying plugin %q: %v

What it means

CreateSnapshot validates each volume's CSI plugin by querying the state store via CSIPluginByID. When that lookup returns an error (state store failure), the error is appended to a multierror and the volume is skipped. This is not a plugin-missing error — it means the query itself failed, typically due to a snapshot of the state store or an internal store error.

Source

Thrown at nomad/csi_endpoint.go:1635

		vol, err := state.CSIVolumeByID(nil, args.RequestNamespace(), snap.SourceVolumeID)
		if err != nil {
			multierror.Append(&mErr, fmt.Errorf("error querying volume %q: %v", snap.SourceVolumeID, err))
			continue
		}
		if vol == nil {
			multierror.Append(&mErr, fmt.Errorf("no such volume %q", snap.SourceVolumeID))
			continue
		}

		pluginID := snap.PluginID
		if pluginID == "" {
			pluginID = vol.PluginID
		}

		plugin, err := state.CSIPluginByID(nil, pluginID)
		if err != nil {
			multierror.Append(&mErr,
				fmt.Errorf("error querying plugin %q: %v", pluginID, err))
			continue
		}
		if plugin == nil {
			multierror.Append(&mErr, fmt.Errorf("no such plugin %q", pluginID))
			continue
		}
		if !plugin.HasControllerCapability(structs.CSIControllerSupportsCreateDeleteSnapshot) {
			multierror.Append(&mErr,
				fmt.Errorf("plugin %q does not support snapshot", pluginID))
			continue
		}

		secrets := vol.Secrets
		// merge request secrets onto volume secrets
		maps.Copy(secrets, snap.Secrets)

		cReq := &cstructs.ClientCSIControllerCreateSnapshotRequest{
			ExternalSourceVolumeID: vol.ExternalID,

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check server logs for the underlying state store error returned by CSIPluginByID to identify the real cause
2. Verify Raft cluster health (nomad server members, nomad operator raft list-peers) and that a stable leader exists
3. Restart/replace the failing Nomad server and restore from a known-good state snapshot if corruption is confirmed

// before the snapshot RPC
pluginStatus, err := client.Plugins().Get(pluginID)
if err != nil { return fmt.Errorf("plugin %q not queryable: %w", pluginID, err) }

// retry on transient state-store errors
for i := 0; i < 3; i++ {
    _, err := client.Volumes().CreateSnapshot(req)
    if err == nil { break }
    if !isStateStoreError(err) { return err } // stop on 'no such plugin' etc.
    time.Sleep(backoff(i))
}

When it happens

Trigger: Calling the CreateSnapshot RPC (POST /v1/volumes/snapshot or nomad volume snapshot create) when the server's state store fails to read the CSIPlugin table, e.g. state store corruption or a boltdb/raft error during the read.

Common situations: Degraded Raft state on a server, state store file corruption, or read errors during leader transitions. Rare in practice; most users instead hit 'no such plugin'.

Related errors

• error parsing: root should be an object
• cannot specify Accessor ID
• failed to read dynamic plugin registry state: %v
• error getting plugin: %s, %v
• error querying volume %q: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/57f34f42f3f78872. Report an issue: GitHub.