{"record":{"id":"5a13e3a0d0485b72","repo":"hashicorp/nomad","slug":"error-creating-rpc-client-for-executor-plugin-v","errorCode":null,"errorMessage":"error creating rpc client for executor plugin: %v","messagePattern":"error creating rpc client for executor plugin: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/shared/executor/utils.go","lineNumber":106,"sourceCode":"\t\tPlugins:          GetPluginMap(logger, false, compute),\n\t\tAllowedProtocols: []plugin.Protocol{plugin.ProtocolGRPC},\n\t\tLogger:           logger.Named(\"executor\"),\n\t}\n\texec, pluginClient, err := newExecutorClient(config, logger)\n\tif err != nil {\n\t\treturn nil, nil, err\n\t}\n\tif _, err := exec.Version(); err != nil {\n\t\treturn nil, nil, err\n\t}\n\treturn exec, pluginClient, nil\n}\n\nfunc newExecutorClient(config *plugin.ClientConfig, logger hclog.Logger) (Executor, *plugin.Client, error) {\n\texecutorClient := plugin.NewClient(config)\n\trpcClient, err := executorClient.Client()\n\tif err != nil {\n\t\treturn nil, nil, fmt.Errorf(\"error creating rpc client for executor plugin: %v\", err)\n\t}\n\n\traw, err := rpcClient.Dispense(\"executor\")\n\tif err != nil {\n\t\treturn nil, nil, fmt.Errorf(\"unable to dispense the executor plugin: %v\", err)\n\t}\n\texecutorPlugin, ok := raw.(Executor)\n\tif !ok {\n\t\treturn nil, nil, fmt.Errorf(\"unexpected executor rpc type: %T\", raw)\n\t}\n\treturn executorPlugin, executorClient, nil\n}\n\nfunc processStateToProto(ps *ProcessState) (*proto.ProcessState, error) {\n\ttimestamp, err := ptypes.TimestampProto(ps.Time)\n\tif err != nil {\n\t\treturn nil, err\n\t}","sourceCodeStart":88,"sourceCodeEnd":124,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/shared/executor/utils.go#L88-L124","documentation":"This error is returned by newExecutorClient when hashicorp/go-plugin's plugin.NewClient(...).Client() fails to establish the RPC connection to the launched executor plugin subprocess. It wraps the underlying handshake error, which typically reflects a failure to start or negotiate the plugin protocol (negotiated version mismatch, failed handshake, or the plugin process exiting early).","triggerScenarios":"Called via CreateExecutor (task startup) or ReattachToExecutor (client restart). plugin.NewClient spawns the executor binary (executor.main or logmon/plugin binaries via exec.Cmd) over a unix socket or TCP port and performs the go-plugin handshake; if the subprocess dies before responding or the handshake times out, Client() returns an error.","commonSituations":"Executor binary path wrong or deleted (plugin.LogLevel/env not propagated); plugin subprocess crashes on startup (OOM, missing cgroup, seccomp killing it); negotiated protocol version mismatch after a Nomad client/server version skew; sandbox environments blocking unix socket creation; PLUGIN_UNIX_SOCKET_DIR permission problems.","solutions":["Read the wrapped %v error: 'connection refused'/'no such file' means the plugin subprocess died — check client logs just before this error for the executor's own stderr","Verify the executor binary exists and is executable in the plugin_dir on the client, and that client/plugin versions match the server","Check for sandbox restrictions (seccomp, AppArmor, lack of /tmp sockets) that prevent go-plugin's unix socket handshake","If it reproduces on reattach, the original executor process is gone; verify the executor pidfile/process is alive before ReattachToExecutor","Upgrade Nomad: older versions had handshake race fixes in go-plugin"],"exampleFix":"// before: blindly retrying task start without inspecting the wrapped cause\nif _, _, err := CreateExecutor(...); err != nil {\n    return err\n}\n// after: log and surface the underlying go-plugin handshake cause\nif _, _, err := CreateExecutor(...); err != nil {\n    logger.Error(\"executor plugin failed\", \"cause\", err)\n    return fmt.Errorf(\"create executor: %w\", err)\n}","handlingStrategy":"retry","validationCode":"// before launching, ensure the plugin binary is present and executable\nif info, err := os.Stat(executorBinary); err != nil || info.Mode()&0111 == 0 {\n    return fmt.Errorf(\"executor binary missing or not executable: %s\", executorBinary)\n}","typeGuard":null,"tryCatchPattern":"executor, client, err := CreateExecutor(...)\nif err != nil {\n    logger.Error(\"executor rpc client failed\", \"err\", err)\n    // retry once after delay; go-plugin handshake failures are often transient (slow subprocess start)\n    time.Sleep(2 * time.Second)\n    executor, client, err = CreateExecutor(...)\n    if err != nil { return fmt.Errorf(\"create executor: %w\", err) }\n}","preventionTips":["Keep client and server Nomad versions identical to avoid protocol skew","Verify plugin_dir contents and executor binary executability in provisioning","Monitor executor subprocess stderr in client logs for early crashes","Avoid sandboxes/seccomp profiles that block unix socket creation in the plugin handshake"],"tags":["go-plugin","rpc","plugin-handshake","nomad"],"backgroundTag":"plugin-handshake-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}