hashicorp/nomad · error

plugin missing controller: %s

Error message

plugin missing controller: %s

What it means

CSIPlugin.AddPlugin returns this when it is about to update the controller entry for nodeID and finds the map says a controller exists ('ok' is true) but the stored pointer is nil. This internally inconsistent state (map key present, value nil) prevents a safe health decrement, so it surfaces an error instead of panicking.

Source

Thrown at nomad/structs/csi.go:1339

			return c.NodeInfo.SupportsExpand
		case CSINodeSupportsCondition:
			return c.NodeInfo.SupportsCondition
		default:
			return false
		}
	}
	return false
}

// AddPlugin adds a single plugin running on the node. Called from state.NodeUpdate in a
// transaction
func (p *CSIPlugin) AddPlugin(nodeID string, info *CSIInfo) error {
	if info.ControllerInfo != nil {
		p.ControllerRequired = info.RequiresControllerPlugin
		prev, ok := p.Controllers[nodeID]
		if ok {
			if prev == nil {
				return fmt.Errorf("plugin missing controller: %s", nodeID)
			}
			if prev.Healthy {
				p.ControllersHealthy -= 1
			}
		}

		// note: for this to work as expected, only a single
		// controller for a given plugin can be on a given Nomad
		// client, they also conflict on the client so this should be
		// ok
		if prev != nil || info.Healthy {
			p.Controllers[nodeID] = info
		}
		if info.Healthy {
			p.ControllersHealthy += 1
		}
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Restart the Nomad client agent so it re-fingerprints the plugin and repopulates Controllers with a valid CSIInfo.
2. Run nomad plugin status <plugin-id>; if the plugin is half-registered, deregister and let the client re-register (delete the plugin or stop/start the plugin task).
3. If the client keeps failing, remove the stale client node (nomad node eligibility -disable / stop client) so the plugin entry is rebuilt.
4. Report/pin down the writer that set the nil controller (check nomad agent logs around 'csi') — it is a state-consistency bug, not a user configuration error.

Example fix

// before
// map entry tombstoned to nil by earlier teardown
p.Controllers[nodeID] = nil
// after
// delete the key instead of storing nil so the map never claims a missing controller
delete(p.Controllers, nodeID)
p.ControllersHealthy = 0

prev, ok := p.Controllers[nodeID]
if ok && prev == nil {
    // repair instead of erroring: drop the nil tombstone
    delete(p.Controllers, nodeID)
    prev, ok = nil, false
}
_ = prev
_ = ok

func hasController(p *structs.CSIPlugin, nodeID string) bool {
    c, ok := p.Controllers[nodeID]
    return ok && c != nil
}

err := plugin.AddPlugin(nodeID, info)
if err != nil {
    if strings.Contains(err.Error(), "plugin missing controller") {
        // state inconsistency: restart client or deregister/re-register the plugin
        return reconcilePlugin(pluginID)
    }
    return err
}

When it happens

Trigger: A CSI plugin fingerprint update (Node.UpdateCSIPlugin RPC) for a node whose Controllers map contains a nil-valued entry for that nodeID while info.ControllerInfo != nil — e.g. after partial de-registration, a nil overwriting bug, or an incomplete state restore.

Common situations: A plugin transitions from controller-present to absent and back while the fingerprint loop fires concurrently; plugin data deserialized from raft where the controller was tombstoned to nil; region/agent restart with partially flushed CSI plugin state.

Related errors

• plugin missing node: %s
• could not validate task driver capabilities: %v
• controller attach volume: %v
• controller validate volume: %v
• controller detach volume: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/09868031e4da182e. Report an issue: GitHub.