hashicorp/nomad · error

plugin not found: %s

Error message

plugin not found: %s

What it means

volAndPluginLookup found the CSI volume but the CSI plugin that owns it (vol.PluginID) is missing from the state store, returning "plugin not found: <id>". For controller-required volumes Nomad needs the plugin to send controller RPCs, so a volume whose plugin has been deregistered or whose plugin tasks are gone cannot be used. This only fires for volumes with ControllerRequired=true.

Source

Thrown at nomad/csi_endpoint.go:622

	if err != nil {
		return nil, nil, err
	}
	if vol == nil {
		return nil, nil, fmt.Errorf("volume not found: %s", volID)
	}
	if !vol.ControllerRequired {
		return nil, vol, nil
	}

	// note: we do this same lookup in CSIVolumeByID but then throw
	// away the pointer to the plugin rather than attaching it to
	// the volume so we have to do it again here.
	plug, err := state.CSIPluginByID(nil, vol.PluginID)
	if err != nil {
		return nil, nil, err
	}
	if plug == nil {
		return nil, nil, fmt.Errorf("plugin not found: %s", vol.PluginID)
	}
	return plug, vol, nil
}

// serializedControllerRPC ensures we're only sending a single controller RPC to
// a given plugin if the RPC can cause conflicting state changes.
//
// The CSI specification says that we SHOULD send no more than one in-flight
// request per *volume* at a time, with an allowance for losing state
// (ex. leadership transitions) which the plugins SHOULD handle gracefully.
//
// In practice many CSI plugins rely on k8s-specific sidecars for serializing
// storage provider API calls globally (ex. concurrently attaching EBS volumes
// to an EC2 instance results in a race for device names). So we have to be much
// more conservative about concurrency in Nomad than the spec allows.
func (v *CSIVolume) serializedControllerRPC(pluginID string, fn func() error) error {

	for {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check `nomad plugin status`; re-run the CSI plugin job on the client(s) so it re-registers under the same plugin ID.
2. Verify the volume's plugin_id in its registration spec matches the ID the plugin job advertises (`plugin_id` field in the task's csi_plugin block).
3. If the plugin is permanently gone, deregister the orphaned volumes and register them against the correct plugin.
4. Ensure at least one healthy node runs the plugin task so it doesn't get GC'd again (`nomad plugin status <id>` shows healthy nodes).

Example fix

hcl
# before: plugin task removed from client job
csi_plugin { id = "aws.ebs" type = "node" }
# after: run node+controller so the plugin stays registered
csi_plugin { id = "aws.ebs" type = "node" mount_dir = "/csi" }
# and a controller instance with type = "controller"

plug, _, err := client.CSIPlugins().Info(vol.PluginID, nil)
if err != nil || plug == nil {
    return fmt.Errorf("plugin %s missing; deploy the CSI plugin job first", vol.PluginID)
}
if len(plug.Nodes.HealthyNodes) == 0 || (vol.ControllerRequired && plug.Controllers.HealthyControllers == 0) {
    return fmt.Errorf("plugin %s has no healthy instances", vol.PluginID)
}

if err != nil && strings.Contains(err.Error(), "plugin not found") {
    // redeploy the CSI plugin job, then retry
}

When it happens

Trigger: Raised when state.CSIPluginByID(nil, vol.PluginID) returns nil during controllerPublishVolume or Delete: the plugin was deregistered (`nomad plugin deregister`), all plugin client tasks were stopped so the plugin GC'd, or the plugin job was removed while its volumes remain registered.

Common situations: The CSI plugin job (e.g. aws-ebs-controller/Node) was stopped or failed and Nomad garbage-collected the plugin; cluster migration where the plugin job wasn't redeployed before volumes were claimed; `nomad system gc` removing an unhealthy plugin.

Related errors

• controller attach volume: %v
• controller validate volume: %v
• controller detach volume: %v
• controller create volume: %v
• controller expand volume: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/2af8a36e45869603. Report an issue: GitHub.