hashicorp/nomad · error

failed to find instances of controller plugin %q

Error message

failed to find instances of controller plugin %q

What it means

If the CSI plugin record exists but its Controllers map is empty, clientIDsForController returns 'failed to find instances of controller plugin %q'. The plugin is registered but no controller-capable instance has ever reported to it, so there is nowhere to forward the controller RPC.

Source

Thrown at nomad/client_csi_endpoint.go:312

	}

	ws := memdb.NewWatchSet()

	// note: plugin IDs are not scoped to region but volumes are. so any Nomad
	// client we get for a controller is already in the same region for the
	// volume.
	plugin, err := snap.CSIPluginByID(ws, pluginID)
	if err != nil {
		return nil, fmt.Errorf("error getting plugin: %s, %v", pluginID, err)
	}
	if plugin == nil {
		return nil, fmt.Errorf("plugin missing: %s", pluginID)
	}

	clientIDs := []string{}

	if len(plugin.Controllers) == 0 {
		return nil, fmt.Errorf("failed to find instances of controller plugin %q", pluginID)
	}

	var merr error
	for clientID, controller := range plugin.Controllers {
		if !controller.IsController() {
			// we don't have separate types for CSIInfo depending on whether
			// it's a controller or node. this error should never make it to
			// production
			merr = errors.Join(merr, fmt.Errorf(
				"plugin instance %q is not a controller but was registered as one - this is always a bug", controller.AllocID))
			continue
		}

		if !controller.Healthy {
			merr = errors.Join(merr, fmt.Errorf(
				"plugin instance %q is not healthy", controller.AllocID))
			continue
		}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the controller plugin job: `nomad job status <csi-plugin-job>` and fix scheduling failures or restart it.
2. Verify node constraints/CSIPluginID placement so the controller can be scheduled.
3. Confirm the driver actually ships a controller component (some drivers are node-only).
4. Once `nomad plugin status <id>` shows controller instances healthy, retry.

Example fix

# before: controller count 0
nomad job stop aws-ebs-csi-controller
# after
nomad job run aws-ebs-csi-controller.nomad.hcl
nomad plugin status aws-ebs-controller  # Controllers: 1/1 healthy

p, _, _ := client.Plugins().Info(ctx, pluginID, nil)
if p != nil && p.ControllersExpected > 0 && p.ControllersHealthy == 0 {
    return fmt.Errorf("controller plugin %q has no running instances", pluginID)
}

func hasControllerInstances(p *api.CSIPlugin) bool {
    return p != nil && p.ControllersHealthy > 0
}

err := csi.ControllerListVolumes(args, reply)
if err != nil && strings.Contains(err.Error(), "failed to find instances of controller plugin") {
    return fmt.Errorf("start the controller plugin job: %w", err)
}

When it happens

Trigger: Controller RPC sent when the controller plugin job has zero running allocations (controller-only job dead, or only node plugins running for a plugin that also needs a controller).

Common situations: Controller plugin job stopped/scaled to zero; plugin job failing to schedule (unsatisfiable constraints, no eligible clients); a node-only CSI plugin mistakenly used as a controller target.

Related errors

• cannot upload management tokens
• CSI plugin failed to register: %w
• plugin missing: %s
• no such plugin %q
• error parsing: root should be an object

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/47a5328e0b8e673e. Report an issue: GitHub.