What causes "%s plugin '%s' did not become ready: %w" in hashicorp/nomad?

Check nomad agent logs for the CSI plugin supervisor and the plugin container logs for startup failures

%s plugin '%s' did not become ready: %w

Error message

%s plugin '%s' did not become ready: %w

What it means

csiManager.WaitForPlugin waits up to one minute for a CSI plugin (controller or node type) to register with the plugin registry, then lazily creates its volume manager instance. This error wraps the registry's failure — usually context deadline exceeded because the plugin never became ready within the minute.

Source

Thrown at client/pluginmanager/csimanager/manager.go:88

	updateNodeCSIInfoFunc UpdateNodeCSIInfoFunc

	shutdownCtx         context.Context
	shutdownCtxCancelFn context.CancelFunc
	shutdownCh          chan struct{}
}

func (c *csiManager) PluginManager() pluginmanager.PluginManager {
	return c
}

// WaitForPlugin waits for a specific plugin to be registered and available,
// unless the context is canceled, or it takes longer than a minute.
func (c *csiManager) WaitForPlugin(ctx context.Context, pType, pID string) error {
	ctx, cancel := context.WithTimeout(ctx, time.Minute)
	defer cancel()
	p, err := c.registry.WaitForPlugin(ctx, pType, pID)
	if err != nil {
		return fmt.Errorf("%s plugin '%s' did not become ready: %w", pType, pID, err)
	}
	c.instancesLock.Lock()
	defer c.instancesLock.Unlock()
	c.ensureInstance(p)
	return nil
}

func (c *csiManager) ManagerForPlugin(ctx context.Context, pluginID string) (VolumeManager, error) {
	c.instancesLock.RLock()
	defer c.instancesLock.RUnlock()
	nodePlugins, hasAnyNodePlugins := c.instances["csi-node"]
	if !hasAnyNodePlugins {
		return nil, fmt.Errorf("no storage node plugins found")
	}

	mgr, hasPlugin := nodePlugins[pluginID]
	if !hasPlugin {
		return nil, fmt.Errorf("plugin %s for type csi-node not found", pluginID)

View on GitHub (pinned to 482b49bf1a)

Solutions

Check nomad agent logs for the CSI plugin supervisor and the plugin container logs for startup failures
Verify the task running the CSI plugin is healthy (nomad job status / alloc events) and that its socket is registered on the node
Confirm pType is one of "csi-controller"/"csi-node" and pID matches the plugin ID exactly
Fix plugin deployment issues (image, capabilities, host volumes, csi_plugin config) so registration completes; increase surrounding context timeout only if plugin startup legitimately exceeds one minute
Retry after fixing; ensure the node has healthy CSI plugins before scheduling workloads that mount volumes

Example fix

// before: failing silently when plugin not ready within 1m
if err := client.WaitForPlugin(ctx, "csi-node", pluginID); err != nil {
    return err
}

// after: pre-check plugin health and use a deadline-aware context
plugins, err := node.CSIMutex... // or check alloc health first
if !csiPluginHealthy(pluginID) {
    return fmt.Errorf("CSI plugin %s unhealthy; fix plugin task before mounting", pluginID)
}
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
defer cancel()
if err := client.WaitForPlugin(ctx, "csi-node", pluginID); err != nil {
    return fmt.Errorf("wait for CSI plugin %s: %w", pluginID, err)
}

Defensive patterns

Strategy: retry

Validate before calling

// Pre-check that a CSI plugin task is running before waiting
allocs, _ := client.Jobs().Allocations("csi-plugin-job", nil, nil)
healthy := false
for _, a := range allocs {
	if a.ClientStatus == "running" {
		healthy = true
	}
}
if !healthy {
	return fmt.Errorf("CSI plugin job not running; fix deployment before waiting")
}

Try / catch

err := client.WaitForPlugin(ctx, pType, pID)
if err != nil {
	if strings.Contains(err.Error(), "did not become ready") {
		// bounded retry with backoff; plugin may still be starting
		return retryWithContext(ctx, 3, 10*time.Second, func() error {
			return client.WaitForPlugin(ctx, pType, pID)
		})
	}
	return err
}

Prevention

Ensure CSI plugin tasks are healthy and constrained to nodes where volumes will mount
Check plugin container logs and csi_plugin stanza (mount_linux/mount_darwin, type) for misconfig
Keep the surrounding context deadline longer than expected plugin startup
Use nomad node status -verbose to confirm the plugin ID and health before waiting

When it happens

Trigger: Calling WaitForPlugin(ctx, pType, pID) (via client.CSIVolumes or volume unmount paths) when the CSI plugin container failed to start, crashed, did not register its socket with the Nomad plugin supervisor, or is slower than 60 seconds to become ready.

Common situations: A CSI plugin deployment whose Docker image is wrong or failing health checks; node lacks the plugin's required privileges/capabilities (e.g. mount caps); plugin socket handshake failing; slow storage backend making plugin init exceed a minute; wrong pType/pID spelling.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/bc985dda4251fe66. Report an issue: GitHub.