What causes "error creating rpc client for executor plugin: %v" in hashicorp/nomad?

Read the wrapped %v error: 'connection refused'/'no such file' means the plugin subprocess died — check client logs just before this error for the executor's own stderr

hashicorp/nomad · error

error creating rpc client for executor plugin: %v

Error message

error creating rpc client for executor plugin: %v

What it means

This error is returned by newExecutorClient when hashicorp/go-plugin's plugin.NewClient(...).Client() fails to establish the RPC connection to the launched executor plugin subprocess. It wraps the underlying handshake error, which typically reflects a failure to start or negotiate the plugin protocol (negotiated version mismatch, failed handshake, or the plugin process exiting early).

Source

Thrown at drivers/shared/executor/utils.go:106

		Plugins:          GetPluginMap(logger, false, compute),
		AllowedProtocols: []plugin.Protocol{plugin.ProtocolGRPC},
		Logger:           logger.Named("executor"),
	}
	exec, pluginClient, err := newExecutorClient(config, logger)
	if err != nil {
		return nil, nil, err
	}
	if _, err := exec.Version(); err != nil {
		return nil, nil, err
	}
	return exec, pluginClient, nil
}

func newExecutorClient(config *plugin.ClientConfig, logger hclog.Logger) (Executor, *plugin.Client, error) {
	executorClient := plugin.NewClient(config)
	rpcClient, err := executorClient.Client()
	if err != nil {
		return nil, nil, fmt.Errorf("error creating rpc client for executor plugin: %v", err)
	}

	raw, err := rpcClient.Dispense("executor")
	if err != nil {
		return nil, nil, fmt.Errorf("unable to dispense the executor plugin: %v", err)
	}
	executorPlugin, ok := raw.(Executor)
	if !ok {
		return nil, nil, fmt.Errorf("unexpected executor rpc type: %T", raw)
	}
	return executorPlugin, executorClient, nil
}

func processStateToProto(ps *ProcessState) (*proto.ProcessState, error) {
	timestamp, err := ptypes.TimestampProto(ps.Time)
	if err != nil {
		return nil, err
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

Read the wrapped %v error: 'connection refused'/'no such file' means the plugin subprocess died — check client logs just before this error for the executor's own stderr
Verify the executor binary exists and is executable in the plugin_dir on the client, and that client/plugin versions match the server
Check for sandbox restrictions (seccomp, AppArmor, lack of /tmp sockets) that prevent go-plugin's unix socket handshake
If it reproduces on reattach, the original executor process is gone; verify the executor pidfile/process is alive before ReattachToExecutor
Upgrade Nomad: older versions had handshake race fixes in go-plugin

Example fix

// before: blindly retrying task start without inspecting the wrapped cause
if _, _, err := CreateExecutor(...); err != nil {
    return err
}
// after: log and surface the underlying go-plugin handshake cause
if _, _, err := CreateExecutor(...); err != nil {
    logger.Error("executor plugin failed", "cause", err)
    return fmt.Errorf("create executor: %w", err)
}

Defensive patterns

Strategy: retry

Validate before calling

// before launching, ensure the plugin binary is present and executable
if info, err := os.Stat(executorBinary); err != nil || info.Mode()&0111 == 0 {
    return fmt.Errorf("executor binary missing or not executable: %s", executorBinary)
}

Try / catch

executor, client, err := CreateExecutor(...)
if err != nil {
    logger.Error("executor rpc client failed", "err", err)
    // retry once after delay; go-plugin handshake failures are often transient (slow subprocess start)
    time.Sleep(2 * time.Second)
    executor, client, err = CreateExecutor(...)
    if err != nil { return fmt.Errorf("create executor: %w", err) }
}

Prevention

Keep client and server Nomad versions identical to avoid protocol skew
Verify plugin_dir contents and executor binary executability in provisioning
Monitor executor subprocess stderr in client logs for early crashes
Avoid sandboxes/seccomp profiles that block unix socket creation in the plugin handshake

When it happens

Trigger: Called via CreateExecutor (task startup) or ReattachToExecutor (client restart). plugin.NewClient spawns the executor binary (executor.main or logmon/plugin binaries via exec.Cmd) over a unix socket or TCP port and performs the go-plugin handshake; if the subprocess dies before responding or the handshake times out, Client() returns an error.

Common situations: Executor binary path wrong or deleted (plugin.LogLevel/env not propagated); plugin subprocess crashes on startup (OOM, missing cgroup, seccomp killing it); negotiated protocol version mismatch after a Nomad client/server version skew; sandbox environments blocking unix socket creation; PLUGIN_UNIX_SOCKET_DIR permission problems.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/5a13e3a0d0485b72. Report an issue: GitHub.