hashicorp/nomad · error

csi_plugin lookup error: %s %v

Error message

csi_plugin lookup error: %s %v

What it means

Wraps a memdb lookup failure when fetching a CSIPlugin row by ID while registering a node's CSI plugins (upsertCSIPluginsForNode). The First() call on TableCSIPlugins returned an error, meaning the transaction or index is in a bad state — not merely that the plugin is missing (that yields raw == nil).

Source

Thrown at nomad/state/state_store.go:1415

		e.CreateIndex = index
		node.Events = append(node.Events, e)
	}

	// Keep node events pruned to not exceed the max allowed
	if l := len(node.Events); l > structs.MaxRetainedNodeEvents {
		delta := l - structs.MaxRetainedNodeEvents
		node.Events = node.Events[delta:]
	}
}

// upsertCSIPluginsForNode indexes csi plugins for volume retrieval, with health. It's called
// on upsertNodeEvents, so that event driven health changes are updated
func upsertCSIPluginsForNode(txn *txn, node *structs.Node, index uint64) error {

	upsertFn := func(info *structs.CSIInfo) error {
		raw, err := txn.First(TableCSIPlugins, "id", info.PluginID)
		if err != nil {
			return fmt.Errorf("csi_plugin lookup error: %s %v", info.PluginID, err)
		}

		var plug *structs.CSIPlugin
		if raw != nil {
			plug = raw.(*structs.CSIPlugin).Copy()
		} else {
			if !info.Healthy {
				// we don't want to create new plugins for unhealthy
				// allocs, otherwise we'd recreate the plugin when we
				// get the update for the alloc becoming terminal
				return nil
			}
			plug = structs.NewCSIPlugin(info.PluginID, index)
		}

		// the plugin may have been created by the job being updated, in which case
		// this data will not be configured, it's only available to the fingerprint
		// system

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry node registration — the error is typically transient state corruption during the transaction.
2. Check the wrapped error (%v suffix) for the actual memdb cause and address it specifically.
3. Confirm the client's CSI fingerprint data (plugin IDs) are well-formed; malformed IDs can trip index lookups.
4. Upgrade Nomad if on an older release with known CSI state-store bugs.

Example fix

// before: plugin ID comes straight from fingerprint
info.PluginID
// after: validate before registration on the client side
if info.PluginID == "" {
	return fmt.Errorf("csi plugin fingerprint missing PluginID")
}

// client side: validate fingerprinted CSI plugins before registration
for _, info := range node.CSIControllerPlugins {
    if info.PluginID == "" { return errors.New("CSI plugin fingerprint missing PluginID") }
}

if err != nil && strings.Contains(err.Error(), "csi_plugin lookup error") {
    logger.Warn("csi plugin lookup failed during registration; retrying", "err", err)
    return retryNodeRegister(node)
}

When it happens

Trigger: Node registration (upsertNodeTxn) where the node's fingerprint includes CSIInfo entries and txn.First(TableCSIPlugins, "id", pluginID) errors, e.g. invalid index name or a failed transaction.

Common situations: Rarely hit in practice; appears in logs when a Nomad server's in-memory state is inconsistent, after a failed upgrade/migration, or in bug reports involving CSI plugin registration.

Related errors

• csi_plugins insert error: %v
• csi_plugins lookup failed: %v
• csi_plugins lookup error %s: %v
• csi_plugins update error %s: %v
• volume insert: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/46641f4243bc3ec8. Report an issue: GitHub.