hashicorp/nomad · error

CSI plugin failed probe: %w

Error message

CSI plugin failed probe: %w

What it means

ensureSupervisorLoop runs a background probe loop after the CSI plugin task starts; if the plugin fails its health probe (supervisorLoopOnce returns an error or unhealthy) while the start context is done, the hook calls h.restartTask with 'CSI plugin failed probe: %w', tearing down and restarting the plugin task. It signals that the plugin container started but never became a healthy CSI plugin within expectations.

Source

Thrown at client/allocrunner/taskrunner/plugin_supervisor_hook.go:302

		h.supervisorIsRunningLock.Lock()
		h.supervisorIsRunning = false
		client.Close()
		supervisorCtxCancel()
		startCancelFn()
		h.supervisorIsRunningLock.Unlock()
	}()

	t := time.NewTimer(0)

	var err error
	var pluginHealthy bool

	// Step 1: Wait for the plugin to initially become available.
WAITFORREADY:
	for {
		select {
		case <-startCtx.Done():
			h.restartTask(ctx, fmt.Errorf("CSI plugin failed probe: %w", err))
			return
		case <-supervisorCtx.Done():
			return
		case <-t.C:
			pluginHealthy, err = h.supervisorLoopOnce(startCtx, client)
			if err != nil || !pluginHealthy {
				h.logger.Debug("CSI plugin not ready", "error", err)
				// Use only a short delay here to optimize for quickly
				// bringing up a plugin
				t.Reset(5 * time.Second)
				continue
			}

			// Mark the plugin as healthy in a task event
			h.logger.Debug("CSI plugin is ready")
			h.previousHealthState = pluginHealthy
			event := structs.NewTaskEvent(structs.TaskPluginHealthy)
			event.SetMessage(fmt.Sprintf("plugin: %s", h.task.CSIPluginConfig.ID))

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped %w error and plugin task logs (nomad alloc logs) to find why the probe failed
2. Verify the plugin's socket appears in the socket mount point and the plugin binary args match the provider's requirements
3. Check the task's resource limits and host privileges (Linux capabilities, mount propagation) required by the CSI plugin
4. Nomad will restart the task automatically; if it crash-loops, pin a known-good plugin image/version compatible with your Nomad release

Example fix

// before
task "plugin" {
  driver = "docker"
  config {
    image = "custom-csi:latest"
    args = ["--endpoint=${CSI_ENDPOINT}"]
  }
}
// after
task "plugin" {
  driver = "docker"
  config {
    image = "custom-csi:v1.5.0"
    args = ["--endpoint=unix://${CSI_ENDPOINT}", "--nodeid=${node.unique.id}"]
    cap_add = ["SYS_ADMIN"]
  }
}

// verify plugin socket appears before assuming crash
ls -l /path/to/socketMountPoint/  # plugin socket must exist shortly after start

// supervisor pattern: bounded restarts with backoff
for attempt := 0; attempt < maxRestarts; attempt++ {
    if healthy := probe(ctx, client); healthy { break }
    time.Sleep(backoff(attempt))
}

When it happens

Trigger: The probe's gRPC GetPluginInfo/GetPluginCapabilities calls fail or report unhealthy — plugin binary crashed on startup, socket never created in the mount point, plugin container OOM-killed, or wrongcsi plugin type/capabilities

Common situations: Docker image with a plugin that exits immediately due to bad args, missing privileges/mounts for the plugin task, storage provider version incompatibility, socket path mismatch between plugin and Nomad config, or host resource exhaustion

Related errors

• CSI Plugin loaded incorrectly
• CSI.ControllerValidateVolume: PluginID is required
• could not validate task driver capabilities: %v
• CSI.ControllerValidateVolume: %w: %v (wraps ErrCSIClientRPCR
• CSI.ControllerAttachVolume: %w: %v (wraps ErrCSIClientRPCRet

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/a9cc2d149a406e3c. Report an issue: GitHub.