{"record":{"id":"a9cc2d149a406e3c","repo":"hashicorp/nomad","slug":"csi-plugin-failed-probe-w","errorCode":null,"errorMessage":"CSI plugin failed probe: %w","messagePattern":"CSI plugin failed probe: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/allocrunner/taskrunner/plugin_supervisor_hook.go","lineNumber":302,"sourceCode":"\t\th.supervisorIsRunningLock.Lock()\n\t\th.supervisorIsRunning = false\n\t\tclient.Close()\n\t\tsupervisorCtxCancel()\n\t\tstartCancelFn()\n\t\th.supervisorIsRunningLock.Unlock()\n\t}()\n\n\tt := time.NewTimer(0)\n\n\tvar err error\n\tvar pluginHealthy bool\n\n\t// Step 1: Wait for the plugin to initially become available.\nWAITFORREADY:\n\tfor {\n\t\tselect {\n\t\tcase <-startCtx.Done():\n\t\t\th.restartTask(ctx, fmt.Errorf(\"CSI plugin failed probe: %w\", err))\n\t\t\treturn\n\t\tcase <-supervisorCtx.Done():\n\t\t\treturn\n\t\tcase <-t.C:\n\t\t\tpluginHealthy, err = h.supervisorLoopOnce(startCtx, client)\n\t\t\tif err != nil || !pluginHealthy {\n\t\t\t\th.logger.Debug(\"CSI plugin not ready\", \"error\", err)\n\t\t\t\t// Use only a short delay here to optimize for quickly\n\t\t\t\t// bringing up a plugin\n\t\t\t\tt.Reset(5 * time.Second)\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\t// Mark the plugin as healthy in a task event\n\t\t\th.logger.Debug(\"CSI plugin is ready\")\n\t\t\th.previousHealthState = pluginHealthy\n\t\t\tevent := structs.NewTaskEvent(structs.TaskPluginHealthy)\n\t\t\tevent.SetMessage(fmt.Sprintf(\"plugin: %s\", h.task.CSIPluginConfig.ID))","sourceCodeStart":284,"sourceCodeEnd":320,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/allocrunner/taskrunner/plugin_supervisor_hook.go#L284-L320","documentation":"ensureSupervisorLoop runs a background probe loop after the CSI plugin task starts; if the plugin fails its health probe (supervisorLoopOnce returns an error or unhealthy) while the start context is done, the hook calls h.restartTask with 'CSI plugin failed probe: %w', tearing down and restarting the plugin task. It signals that the plugin container started but never became a healthy CSI plugin within expectations.","triggerScenarios":"The probe's gRPC GetPluginInfo/GetPluginCapabilities calls fail or report unhealthy — plugin binary crashed on startup, socket never created in the mount point, plugin container OOM-killed, or wrongcsi plugin type/capabilities","commonSituations":"Docker image with a plugin that exits immediately due to bad args, missing privileges/mounts for the plugin task, storage provider version incompatibility, socket path mismatch between plugin and Nomad config, or host resource exhaustion","solutions":["Read the wrapped %w error and plugin task logs (nomad alloc logs) to find why the probe failed","Verify the plugin's socket appears in the socket mount point and the plugin binary args match the provider's requirements","Check the task's resource limits and host privileges (Linux capabilities, mount propagation) required by the CSI plugin","Nomad will restart the task automatically; if it crash-loops, pin a known-good plugin image/version compatible with your Nomad release"],"exampleFix":"// before\ntask \"plugin\" {\n  driver = \"docker\"\n  config {\n    image = \"custom-csi:latest\"\n    args = [\"--endpoint=${CSI_ENDPOINT}\"]\n  }\n}\n// after\ntask \"plugin\" {\n  driver = \"docker\"\n  config {\n    image = \"custom-csi:v1.5.0\"\n    args = [\"--endpoint=unix://${CSI_ENDPOINT}\", \"--nodeid=${node.unique.id}\"]\n    cap_add = [\"SYS_ADMIN\"]\n  }\n}","handlingStrategy":"retry","validationCode":"// verify plugin socket appears before assuming crash\nls -l /path/to/socketMountPoint/  # plugin socket must exist shortly after start","typeGuard":null,"tryCatchPattern":"// supervisor pattern: bounded restarts with backoff\nfor attempt := 0; attempt < maxRestarts; attempt++ {\n    if healthy := probe(ctx, client); healthy { break }\n    time.Sleep(backoff(attempt))\n}","preventionTips":["Pin a known-good CSI plugin image/version for your Nomad release","Grant required capabilities/privileges to the plugin task","Verify plugin args (endpoint, node id) against provider docs","Monitor plugin task logs and resource usage for OOM/crash loops"],"tags":["csi","health-probe","plugin","restart"],"backgroundTag":"csi-plugin-probe-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}