hashicorp/nomad · error

plugin missing node: %s

Error message

plugin missing node: %s

What it means

The node-registry mirror of the controller error: AddPlugin detects the Nodes map contains a nil CSIInfo for nodeID while an update for info.NodeInfo arrives. Since the previous entry is unusable, the function refuses to decrement NodesHealthy and returns this error rather than dereferencing nil.

Source

Thrown at nomad/structs/csi.go:1362

		}

		// note: for this to work as expected, only a single
		// controller for a given plugin can be on a given Nomad
		// client, they also conflict on the client so this should be
		// ok
		if prev != nil || info.Healthy {
			p.Controllers[nodeID] = info
		}
		if info.Healthy {
			p.ControllersHealthy += 1
		}
	}

	if info.NodeInfo != nil {
		prev, ok := p.Nodes[nodeID]
		if ok {
			if prev == nil {
				return fmt.Errorf("plugin missing node: %s", nodeID)
			}
			if prev.Healthy {
				p.NodesHealthy -= 1
			}
		}
		if prev != nil || info.Healthy {
			p.Nodes[nodeID] = info
		}
		if info.Healthy {
			p.NodesHealthy += 1
		}
	}

	return nil
}

// DeleteNode removes all plugins from the node. Called from state.DeleteNode in a
// transaction

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Restart the Nomad client agent to force a clean re-fingerprint of the node plugin.
2. Check nomad plugin status <id>; deregister the broken plugin and let the client re-register it (restart the CSI plugin task).
3. Upgrade Nomad if on an older release — nil-value inconsistencies in CSIPlugin maps were fixed by deleting keys instead of nulling them.
4. As a last resort, drain and remove the affected client node so its plugin entries are fully rebuilt.

Example fix

// before
p.Nodes[nodeID] = nil // leaves key present with nil value
// after
delete(p.Nodes, nodeID)
p.NodesHealthy = 0

prev, ok := p.Nodes[nodeID]
if ok && prev == nil {
    // nil tombstone: remove it before applying the new NodeInfo
    delete(p.Nodes, nodeID)
    prev, ok = nil, false
}
_ = prev
_ = ok

func hasNode(p *structs.CSIPlugin, nodeID string) bool {
    n, ok := p.Nodes[nodeID]
    return ok && n != nil
}

err := plugin.AddPlugin(nodeID, info)
if err != nil {
    if strings.Contains(err.Error(), "plugin missing node") {
        // client-side state is inconsistent; restart the client agent or
        // restart the CSI node plugin task so it re-fingerprints
        return restartCSIPluginTask(nodeID)
    }
    return err
}

When it happens

Trigger: A Node.UpdateCSIPlugin fingerprint update carrying NodeInfo for a node whose Nodes map slot exists but holds nil — typically from a prior deregistration that nulled the value, a concurrent fingerprint update, or corrupted plugin state in raft.

Common situations: CSI node plugin restarting rapidly so teardown and re-fingerprint interleave; snapshot restore where node entries were pruned incompletely; bug in an older Nomad version that set map values to nil instead of deleting keys.

Related errors

• missing external node ID: %v
• plugin missing controller: %s
• could not validate task driver capabilities: %v
• controller attach volume: %v
• controller validate volume: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/cadf486f742b8640. Report an issue: GitHub.