hashicorp/nomad · error

no storage node plugins found

Error message

no storage node plugins found

What it means

ManagerForPlugin looks up an initialized CSI volume manager instance for a plugin ID among the node's registered csi-node plugins. This error is returned when the manager has no csi-node instances at all, meaning no node-type CSI plugin has ever become ready on this client.

Source

Thrown at client/pluginmanager/csimanager/manager.go:101

func (c *csiManager) WaitForPlugin(ctx context.Context, pType, pID string) error {
	ctx, cancel := context.WithTimeout(ctx, time.Minute)
	defer cancel()
	p, err := c.registry.WaitForPlugin(ctx, pType, pID)
	if err != nil {
		return fmt.Errorf("%s plugin '%s' did not become ready: %w", pType, pID, err)
	}
	c.instancesLock.Lock()
	defer c.instancesLock.Unlock()
	c.ensureInstance(p)
	return nil
}

func (c *csiManager) ManagerForPlugin(ctx context.Context, pluginID string) (VolumeManager, error) {
	c.instancesLock.RLock()
	defer c.instancesLock.RUnlock()
	nodePlugins, hasAnyNodePlugins := c.instances["csi-node"]
	if !hasAnyNodePlugins {
		return nil, fmt.Errorf("no storage node plugins found")
	}

	mgr, hasPlugin := nodePlugins[pluginID]
	if !hasPlugin {
		return nil, fmt.Errorf("plugin %s for type csi-node not found", pluginID)
	}

	return mgr.VolumeManager(ctx)
}

// Run starts a plugin manager and should return early
func (c *csiManager) Run() {
	go c.runLoop()
}

func (c *csiManager) runLoop() {
	timer := time.NewTimer(0) // ensure we sync immediately in first pass
	controllerUpdates := c.registry.PluginsUpdatedCh(c.shutdownCtx, "csi-controller")

View on GitHub (pinned to 482b49bf1a)

Solutions

  1. Deploy/ensure the CSI plugin job is running and healthy on the target node (nomad job status <csi-plugin-job>)
  2. Confirm the volume's plugin_id matches a plugin registered on that node via nomad node status <id> -verbose (CSI section)
  3. Check plugin supervisor logs on the client for registration errors
  4. Reschedule or reschedule the workload after the plugin registers; add placement constraints so volumes only run on nodes with their plugin

Example fix

// before: mounting a volume on any node regardless of plugin placement
constraint missing...

// after: constrain workload to nodes that run the plugin
constraint {
  attribute = "${meta.csi_plugin.<plugin_id>.healthy}"
  value     = "true"
}
Defensive patterns

Strategy: fallback

Validate before calling

node, _, _ := client.Nodes().Info(nodeID, nil)
registered := false
for _, p := range node.CSIMutexPlugins {
	if p.Provider == "aws.efs" && p.NodeID != nil {
		registered = true
	}
}
if !registered {
	return fmt.Errorf("no csi-node plugin registered on target node")
}

Try / catch

mgr, err := manager.ManagerForPlugin(ctx, pluginID)
if err != nil {
	if strings.Contains(err.Error(), "no storage node plugins found") {
		// fall back: wait for plugin registration, then retry
		if werr := client.WaitForPlugin(ctx, "csi-node", pluginID); werr != nil {
			return fmt.Errorf("CSI plugin unavailable on node: %w", werr)
		}
		mgr, err = manager.ManagerForPlugin(ctx, pluginID)
	}
	if err != nil {
		return err
	}
}

Prevention

When it happens

Trigger: Requesting a volume manager (e.g. during task volume mounting or GC) before any csi-node plugin registered, or after all CSI plugins were removed from the node.

Common situations: Taskgroup volume references pointing at a host whose CSI plugin task never started; the CSI plugin job is stopped or its alloc failed; a volume is used on a node where the plugin is not deployed; startup ordering where the workload mounts before the plugin fingerprint completes.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/7934f95dc8e44838. Report an issue: GitHub.