hashicorp/nomad · error

failed to probe plugin: %w

Error message

failed to probe plugin: %w

What it means

registerPlugin probes the CSI plugin via client.PluginInfo() to fingerprint its vendor name and version. If that gRPC call returns an error (unreachable socket, timeout, plugin not fully started, CSI spec violation), the error is wrapped as 'failed to probe plugin' and propagates up to ensureSupervisorLoop where it causes a task restart.

Source

Thrown at client/allocrunner/taskrunner/plugin_supervisor_hook.go:381

				}
				h.eventEmitter.EmitEvent(event)
			}

			h.previousHealthState = pluginHealthy

			// This loop is informational and in some plugins this may be expensive to
			// validate. We use a longer timeout (30s) to avoid causing undue work.
			t.Reset(30 * time.Second)
		}
	}
}

func (h *csiPluginSupervisorHook) registerPlugin(client csi.CSIPlugin, socketPath string) (func(), error) {
	// At this point we know the plugin is ready and we can fingerprint it
	// to get its vendor name and version
	info, err := client.PluginInfo()
	if err != nil {
		return nil, fmt.Errorf("failed to probe plugin: %w", err)
	}

	mkInfoFn := func(pluginType string) *dynamicplugins.PluginInfo {
		return &dynamicplugins.PluginInfo{
			Type:    pluginType,
			Name:    h.task.CSIPluginConfig.ID,
			Version: info.PluginVersion,
			ConnectionInfo: &dynamicplugins.PluginConnectionInfo{
				SocketPath: socketPath,
			},
			AllocID: h.alloc.ID,
			Options: map[string]string{
				"Provider":            info.Name, // vendor name
				"MountPoint":          h.mountPoint,
				"ContainerMountPoint": h.task.CSIPluginConfig.StagePublishBaseDir,
			},
		}
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped inner error: gRPC Unavailable means socket/endpoint not ready; deadline means slow startup
2. Ensure the plugin fully implements the CSI Identity service (GetPluginInfo)
3. Give the plugin more time — Nomad probes the socket with a timeout; slow plugins need faster startup or pre-warmed init
4. Verify csi_plugin.mount_dir matches the socket directory the plugin actually creates
5. Check plugin container logs and exit codes for crashes (OOM, missing deps) during startup

Example fix

// before: plugin lacks identity service
case csi.PluginInfo(): return nil // plugin stub never implemented GetPluginInfo
// after: implement CSI Identity service in the plugin
func (p *Plugin) GetPluginInfo(ctx context.Context, _ *csi.GetPluginInfoRequest) (*csi.GetPluginInfoResponse, error) {
    return &csi.GetPluginInfoResponse{Name: "aws-efs", VendorVersion: *version}, nil
}

// probe the plugin manually before Nomad does
conn, err := grpc.Dial("unix://"+socketPath, grpc.WithInsecure(), grpc.WithTimeout(5*time.Second))
if err != nil { log.Fatalf("CSI endpoint not ready: %v", err) }
resp, err := csi.NewIdentityClient(conn).GetPluginInfo(context.Background(), &csi.GetPluginInfoRequest{})

func pluginIdentityOK(resp *csi.GetPluginInfoResponse, err error) bool {
    return err == nil && resp != nil && resp.GetName() != ""
}

if info, err := client.PluginInfo(); err != nil {
    return fmt.Errorf("failed to probe plugin: %w", err)
    // caller: restart with backoff; check wrapped gRPC code
}

When it happens

Trigger: client.PluginInfo() is called on a CSIPlugin client connected to the plugin's Unix socket; the call fails because the socket is not yet accepting connections, the plugin's gRPC server is not serving the Identity service, the probe times out, or the plugin returns a gRPC error.

Common situations: Plugin binary crashes or is OOM-killed right after socket creation; plugin is slow to initialize so the first PluginInfo call races startup; wrong mount_dir/socket path configured; CSI plugin not implementing the Identity GetPluginInfo RPC; network-mode/container isolation hiding the socket.

Related errors

• CSI plugin failed to register: %w
• CSI.ControllerExpandVolume: plugin did not return error or r
• CSI.ControllerDeleteVolume: %v
• CSI.ControllerListVolumes: %v
• %v: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/e55f9a2feb17aae2. Report an issue: GitHub.