hashicorp/nomad · error

csi_plugins lookup error %s: %v

Error message

csi_plugins lookup error %s: %v

What it means

Wraps a memdb First() lookup failure for a CSI plugin ID during deleteNodeCSIPlugins, which cleans plugin health when a node is deregistered (deleteNodeTxn). Note raw == nil (plugin absent) is handled gracefully with continue — this error only fires when the lookup itself errors.

Source

Thrown at nomad/state/state_store.go:1542

// deleteNodeCSIPlugins cleans up CSIInfo node health status, called in DeleteNode
func deleteNodeCSIPlugins(txn *txn, node *structs.Node, index uint64) error {
	if len(node.CSIControllerPlugins) == 0 && len(node.CSINodePlugins) == 0 {
		return nil
	}

	names := map[string]struct{}{}
	for _, info := range node.CSIControllerPlugins {
		names[info.PluginID] = struct{}{}
	}
	for _, info := range node.CSINodePlugins {
		names[info.PluginID] = struct{}{}
	}

	for id := range names {
		raw, err := txn.First(TableCSIPlugins, "id", id)
		if err != nil {
			return fmt.Errorf("csi_plugins lookup error %s: %v", id, err)
		}
		if raw == nil {
			// plugin may have been deregistered but we didn't
			// update the fingerprint yet
			continue
		}

		plug := raw.(*structs.CSIPlugin).Copy()
		err = plug.DeleteNode(node.ID)
		if err != nil {
			return err
		}
		err = updateOrGCPlugin(index, txn, plug)
		if err != nil {
			return err
		}
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry the node delete; the operation is idempotent via Raft replication.
2. Inspect the wrapped error's %v cause for the actual memdb failure.
3. If plugin rows are already inconsistent, use the CSI plugin API to check state and force-deregister stale plugins.
4. Upgrade Nomad if tied to a known CSI deregistration bug.

if err != nil && strings.Contains(err.Error(), "csi_plugins lookup error") {
    logger.Warn("csi plugin lookup failed on node delete; retrying deregistration", "err", err)
    return retryNodeDeregister(nodeID)
}

When it happens

Trigger: DeleteNode / node deregistration where the node carried CSIInfo; iterating the collected plugin IDs and txn.First(TableCSIPlugins, "id", id) returns a memdb error.

Common situations: Seen in server logs when draining or deregistering CSI-capable nodes; points to state-store inconsistency or a Nomad bug rather than operator action.

Related errors

• csi_plugin lookup error: %s %v
• csi_plugins insert error: %v
• csi_plugins lookup failed: %v
• csi_plugins update error %s: %v
• volume insert: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/c1e9db1d03718b21. Report an issue: GitHub.