{"record":{"id":"e55f9a2feb17aae2","repo":"hashicorp/nomad","slug":"failed-to-probe-plugin-w","errorCode":null,"errorMessage":"failed to probe plugin: %w","messagePattern":"failed to probe plugin: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/allocrunner/taskrunner/plugin_supervisor_hook.go","lineNumber":381,"sourceCode":"\t\t\t\t}\n\t\t\t\th.eventEmitter.EmitEvent(event)\n\t\t\t}\n\n\t\t\th.previousHealthState = pluginHealthy\n\n\t\t\t// This loop is informational and in some plugins this may be expensive to\n\t\t\t// validate. We use a longer timeout (30s) to avoid causing undue work.\n\t\t\tt.Reset(30 * time.Second)\n\t\t}\n\t}\n}\n\nfunc (h *csiPluginSupervisorHook) registerPlugin(client csi.CSIPlugin, socketPath string) (func(), error) {\n\t// At this point we know the plugin is ready and we can fingerprint it\n\t// to get its vendor name and version\n\tinfo, err := client.PluginInfo()\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to probe plugin: %w\", err)\n\t}\n\n\tmkInfoFn := func(pluginType string) *dynamicplugins.PluginInfo {\n\t\treturn &dynamicplugins.PluginInfo{\n\t\t\tType:    pluginType,\n\t\t\tName:    h.task.CSIPluginConfig.ID,\n\t\t\tVersion: info.PluginVersion,\n\t\t\tConnectionInfo: &dynamicplugins.PluginConnectionInfo{\n\t\t\t\tSocketPath: socketPath,\n\t\t\t},\n\t\t\tAllocID: h.alloc.ID,\n\t\t\tOptions: map[string]string{\n\t\t\t\t\"Provider\":            info.Name, // vendor name\n\t\t\t\t\"MountPoint\":          h.mountPoint,\n\t\t\t\t\"ContainerMountPoint\": h.task.CSIPluginConfig.StagePublishBaseDir,\n\t\t\t},\n\t\t}\n\t}","sourceCodeStart":363,"sourceCodeEnd":399,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/allocrunner/taskrunner/plugin_supervisor_hook.go#L363-L399","documentation":"registerPlugin probes the CSI plugin via client.PluginInfo() to fingerprint its vendor name and version. If that gRPC call returns an error (unreachable socket, timeout, plugin not fully started, CSI spec violation), the error is wrapped as 'failed to probe plugin' and propagates up to ensureSupervisorLoop where it causes a task restart.","triggerScenarios":"client.PluginInfo() is called on a CSIPlugin client connected to the plugin's Unix socket; the call fails because the socket is not yet accepting connections, the plugin's gRPC server is not serving the Identity service, the probe times out, or the plugin returns a gRPC error.","commonSituations":"Plugin binary crashes or is OOM-killed right after socket creation; plugin is slow to initialize so the first PluginInfo call races startup; wrong mount_dir/socket path configured; CSI plugin not implementing the Identity GetPluginInfo RPC; network-mode/container isolation hiding the socket.","solutions":["Inspect the wrapped inner error: gRPC Unavailable means socket/endpoint not ready; deadline means slow startup","Ensure the plugin fully implements the CSI Identity service (GetPluginInfo)","Give the plugin more time — Nomad probes the socket with a timeout; slow plugins need faster startup or pre-warmed init","Verify csi_plugin.mount_dir matches the socket directory the plugin actually creates","Check plugin container logs and exit codes for crashes (OOM, missing deps) during startup"],"exampleFix":"// before: plugin lacks identity service\ncase csi.PluginInfo(): return nil // plugin stub never implemented GetPluginInfo\n// after: implement CSI Identity service in the plugin\nfunc (p *Plugin) GetPluginInfo(ctx context.Context, _ *csi.GetPluginInfoRequest) (*csi.GetPluginInfoResponse, error) {\n    return &csi.GetPluginInfoResponse{Name: \"aws-efs\", VendorVersion: *version}, nil\n}","handlingStrategy":"retry","validationCode":"// probe the plugin manually before Nomad does\nconn, err := grpc.Dial(\"unix://\"+socketPath, grpc.WithInsecure(), grpc.WithTimeout(5*time.Second))\nif err != nil { log.Fatalf(\"CSI endpoint not ready: %v\", err) }\nresp, err := csi.NewIdentityClient(conn).GetPluginInfo(context.Background(), &csi.GetPluginInfoRequest{})","typeGuard":"func pluginIdentityOK(resp *csi.GetPluginInfoResponse, err error) bool {\n    return err == nil && resp != nil && resp.GetName() != \"\"\n}","tryCatchPattern":"if info, err := client.PluginInfo(); err != nil {\n    return fmt.Errorf(\"failed to probe plugin: %w\", err)\n    // caller: restart with backoff; check wrapped gRPC code\n}","preventionTips":["Make sure the plugin implements CSI Identity GetPluginInfo","Allow the plugin enough init time before registration probes","Verify socket path/mount_dir config matches the plugin's actual socket location","Watch plugin container logs/OOM events; pin stable plugin versions"],"tags":["nomad","csi","grpc","fingerprint","plugin-startup"],"backgroundTag":"csi-plugin-probe-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}