hashicorp/nomad · error

plugin instance %q is not healthy

Error message

plugin instance %q is not healthy

What it means

clientIDsForController skips controller instances whose CSIInfo.Healthy is false, joining 'plugin instance %q is not healthy' into a combined error. If all instances are skipped, the caller has no healthy node to forward the controller RPC to and the joined error surfaces.

Source

Thrown at nomad/client_csi_endpoint.go:327

	clientIDs := []string{}

	if len(plugin.Controllers) == 0 {
		return nil, fmt.Errorf("failed to find instances of controller plugin %q", pluginID)
	}

	var merr error
	for clientID, controller := range plugin.Controllers {
		if !controller.IsController() {
			// we don't have separate types for CSIInfo depending on whether
			// it's a controller or node. this error should never make it to
			// production
			merr = errors.Join(merr, fmt.Errorf(
				"plugin instance %q is not a controller but was registered as one - this is always a bug", controller.AllocID))
			continue
		}

		if !controller.Healthy {
			merr = errors.Join(merr, fmt.Errorf(
				"plugin instance %q is not healthy", controller.AllocID))
			continue
		}

		node, err := getNodeForRpc(snap, clientID)
		if err != nil || node == nil {
			merr = errors.Join(merr, fmt.Errorf(
				"cannot find node %q for plugin instance %q", clientID, controller.AllocID))
			continue
		}

		if node.Status != structs.NodeStatusReady {
			merr = errors.Join(merr, fmt.Errorf(
				"node %q for plugin instance %q is not ready", clientID, controller.AllocID))
			continue
		}

		clientIDs = append(clientIDs, clientID)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect `nomad plugin status <pluginID>` for instance health and the alloc logs of the controller job (`nomad alloc logs <alloc>`).
2. Fix the root cause (OOM, bad credentials, driver probe failure) and redeploy the plugin job.
3. Check `nomad node status` for lost/down clients and restore client agent connectivity so heartbeats resume.
4. Retry once at least one controller instance reports healthy.

Example fix

# before: controller alloc crash-looping on bad secret
# aws_secret_access_key = "stale"
# after: update secrets and reschedule
nomad job run -vault-namespace=csi aws-ebs-csi-controller.nomad.hcl
nomad plugin status aws-ebs-controller  # instances healthy

p, _, _ := client.Plugins().Info(ctx, pluginID, nil)
if p.ControllersHealthy == 0 {
    return fmt.Errorf("wait: all %d controller instances unhealthy", p.ControllersExpected)
}

func anyHealthyController(p *api.CSIPlugin) bool { return p.ControllersHealthy > 0 }

err := csi.ControllerListVolumes(args, reply)
if err != nil && strings.Contains(err.Error(), "is not healthy") {
    // joined error may list several allocs; inspect plugin status then retry
    return retryAfterPluginRecovers(err)
}

When it happens

Trigger: All controller plugin instances on the plugin record have Healthy=false — plugin allocations crashed, failed health checks, or clients stopped heartbeating plugin info after a node loss.

Common situations: Controller alloc OOMing or crashing-looping; nomad client agent down so plugin heartbeats lapse; plugin failed CSI probe after backend credential changes; node drained/terminated while plugin record persists.

Related errors

• %w %s
• error parsing: root should be an object
• missing policy name
• cannot specify Accessor ID
• missing accessor ID

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/da5e996c2d96aa9b. Report an issue: GitHub.