hashicorp/nomad · error

CSI plugin failed to register: %w

Error message

CSI plugin failed to register: %w

What it means

Nomad wraps any error from registerPlugin with this message when the CSI plugin could not be registered into the plugin catalog after the task started. The wrap happens in ensureSupervisorLoop, and the wrapped error triggers restartTask, so the task is restarted in an attempt to recover the plugin registration.

Source

Thrown at client/allocrunner/taskrunner/plugin_supervisor_hook.go:330

				t.Reset(5 * time.Second)
				continue
			}

			// Mark the plugin as healthy in a task event
			h.logger.Debug("CSI plugin is ready")
			h.previousHealthState = pluginHealthy
			event := structs.NewTaskEvent(structs.TaskPluginHealthy)
			event.SetMessage(fmt.Sprintf("plugin: %s", h.task.CSIPluginConfig.ID))
			h.eventEmitter.EmitEvent(event)

			break WAITFORREADY
		}
	}

	// Step 2: Register the plugin with the catalog.
	deregisterPluginFn, err := h.registerPlugin(client, h.socketPath)
	if err != nil {
		h.restartTask(ctx, fmt.Errorf("CSI plugin failed to register: %w", err))
		return
	}
	// De-register plugins on task shutdown
	defer deregisterPluginFn()

	// Step 3: Start the lightweight supervisor loop. At this point,
	// probe failures don't cause the task to restart
	t.Reset(0)
	for {
		select {
		case <-supervisorCtx.Done():
			return
		case <-t.C:
			pluginHealthy, err := h.supervisorLoopOnce(supervisorCtx, client)
			if err != nil {
				h.logger.Error("CSI plugin fingerprinting failed", "error", err)
			}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the plugin container logs to see why the CSI endpoint failed to serve PluginInfo
2. Verify csi_plugin block config (id, type, mount_dir) matches where the plugin exposes its socket
3. Increase liveness/registration timeouts (e.g. plugin stanza's csi_agent_timeout / task resources) if the plugin is slow to start
4. Confirm the socket path is on a shared mount visible to the Nomad client, not an isolated filesystem
5. Ensure the plugin image implements the CSI Identity service (NodePluginInfo/GetPluginInfo)

Example fix

// before
plugin "aws-efs" {
  type = "node"
  mount_dir = "/csi"
}
// after
plugin "aws-efs" {
  type        = "node"
  id          = "aws-efs"
  mount_dir   = "/csi"
  healthchecks { liveness_probe { ... } } # ensure plugin endpoint is actually up
}

// before starting the task, verify the CSI plugin endpoint is reachable
func pluginSocketReady(socketPath string) error {
    _, err := os.Stat(socketPath)
    return err // must exist before registration probes succeed
}

// tolerate and retry registration: restartTask already retries
ticker := time.NewTicker(10 * time.Second)
for attempt := 0; attempt < 5; attempt++ {
    dereg, err := registerPlugin(client, socketPath)
    if err == nil { defer dereg(); break }
    log.Warnf("CSI plugin registration failed, retrying: %v", err)
    <-ticker.C
}

When it happens

Trigger: The CSI plugin task starts and its socket accepts connections, but h.registerPlugin fails — typically client.PluginInfo() errors (timeout, gRPC failure, bad socket path) or the dynamic registration into the catalog fails. ensureSupervisorLoop (called via Poststart) then calls h.restartTask with this wrapped error.

Common situations: Plugin container starts but its CSI endpoint is slow or never ready within the probe timeout; csi_plugin.mount_config/stage/permission stanza misconfiguration causing a wrong socket path; plugin image lacking the CSI identity service; NFS/emptyDir volume where the socket never appears; Nomad client cannot reach the plugin over the Unix socket.

Related errors

• failed to probe plugin: %w
• CSI.ControllerExpandVolume: plugin did not return error or r
• CSI.ControllerDeleteVolume: %v
• CSI.ControllerListVolumes: %v
• plugin missing: %s

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/afb8a1a9a0acd902. Report an issue: GitHub.