hashicorp/nomad · error

csi_plugins lookup failed: %v

Error message

csi_plugins lookup failed: %v

What it means

Wraps a memdb iterator error when scanning the full csi_plugins table during node upsert to detach the client node from plugins no longer running on it. txn.Get(TableCSIPlugins, "id") failed, aborting the node registration transaction.

Source

Thrown at nomad/state/state_store.go:1475

		if err != nil {
			return err
		}
		inUseController[info.PluginID] = struct{}{}
	}

	for _, info := range node.CSINodePlugins {
		err := upsertFn(info)
		if err != nil {
			return err
		}
		inUseNode[info.PluginID] = struct{}{}
	}

	// remove the client node from any plugin that's not
	// running on it.
	iter, err := txn.Get(TableCSIPlugins, "id")
	if err != nil {
		return fmt.Errorf("csi_plugins lookup failed: %v", err)
	}
	for {
		raw := iter.Next()
		if raw == nil {
			break
		}
		plug, ok := raw.(*structs.CSIPlugin)
		if !ok {
			continue
		}
		plug = plug.Copy()

		var hadDelete bool
		if _, ok := inUseController[plug.ID]; !ok {
			if _, asController := plug.Controllers[node.ID]; asController {
				err := plug.DeleteNodeForType(node.ID, structs.CSIPluginTypeController)
				if err != nil {
					return err

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry the node registration; transient iterator failures usually clear on the next attempt.
2. Look at the wrapped %v error for the true memdb cause and fix that root cause.
3. Restart the Nomad server agent if the state store is persistently inconsistent.
4. Upgrade to the latest Nomad patch release for CSI/state-store fixes.

if err != nil && strings.Contains(err.Error(), "csi_plugins lookup failed") {
    logger.Error("state store iteration failed during node upsert", "err", err)
    return backoffRetry(registerNode)
}

When it happens

Trigger: upsertNodeTxn processing a node with CSIInfo; the table iterator acquisition fails (transaction or index state invalid), producing 'csi_plugins lookup failed'.

Common situations: Rare; seen alongside other memdb errors in server logs during state corruption or Nomad bug conditions, typically during node re-registration with CSI plugins attached.

Related errors

• csi_plugin lookup error: %s %v
• csi_plugins insert error: %v
• csi_plugins lookup error %s: %v
• csi_plugins update error %s: %v
• volume insert: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/123e616e0cd1f5df. Report an issue: GitHub.