hashicorp/nomad · error

<remote node streaming RPC error relayed from ack.Error>

Error message

<remote node streaming RPC error relayed from ack.Error>

What it means

NodeStreamingRpc on the Nomad client forwards a streaming RPC to the remote node's agent. When the remote node replies with a Nack-style ack containing a non-empty Error field, the client closes the stream and relays that message verbatim as a new error via errors.New(ack.Error). The '<remote node streaming RPC error relayed from ack.Error>' prefix indicates the real failure happened on another node, not the server handling the request.

Source

Thrown at nomad/client_rpc.go:302

	decoder := codec.NewDecoder(stream, structs.MsgpackHandle)
	header := structs.StreamingRpcHeader{
		Method: method,
	}
	if err := encoder.Encode(header); err != nil {
		stream.Close()
		return nil, err
	}

	// Wait for the acknowledgement
	var ack structs.StreamingRpcAck
	if err := decoder.Decode(&ack); err != nil {
		stream.Close()
		return nil, err
	}

	if ack.Error != "" {
		stream.Close()
		return nil, errors.New(ack.Error)
	}

	return stream, nil
}

// findNodeConnAndForward is a helper for finding the server with a connection
// to the given node and forwarding the RPC to the correct server. This does not
// work for streaming RPCs.
func findNodeConnAndForward(srv *Server, nodeID, method string, args, reply any) error {
	// Determine the Server that has a connection to the node.
	srvWithConn, err := srv.serverWithNodeConn(nodeID, srv.Region())
	if err != nil {
		return err
	}

	if srvWithConn == nil {
		return structs.ErrNoNodeConn
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

  1. Run 'nomad node status <node>' and check the target allocation/task state; retry after confirming the task is running
  2. Verify the client agent on the target node is running and matches the server version (upgrade to align)
  3. Re-run the command against the correct allocation ID if the original task finished
  4. Check the target node's nomad agent logs for the underlying handler error that was relayed

Example fix

// before (command against stale alloc)
nomad alloc exec -task web 1a2b3c ls /
// error: remote node relayed failure
// after: confirm task is running first
nomad alloc status 1a2b3c   # ensure task 'web' is running
nomad alloc exec -task web <current-alloc-id> ls /
Defensive patterns

Strategy: retry

Validate before calling

// shell: verify allocation and task are running before streaming
nomad alloc status <allocID> | grep 'Task .* running' || { echo 'task not running'; exit 1; }

Try / catch

// Go: retry the streaming RPC on relayed errors
var stream *structs.ClientRPCStream
err := retry.Do(func() error {
    s, err := node.NodeStreamingRpc(nodeID, req)
    if err != nil {
        if strings.Contains(err.Error(), "relayed from ack.Error") {
            return retry.TransientError{Err: err} // remote node issue; retry
        }
        return retry.Unrecoverable(err)
    }
    stream = s
    return nil
})

Prevention

When it happens

Trigger: A server calls NodeStreamingRpc to reach a client node (used by exec, stream, logs, and forwardMonitorClient paths); the target node's agent accepts the connection but responds with an ack whose Error field is set, e.g. the exec task is gone, the file does not exist, or the node-side handler rejected the stream.

Common situations: Running 'nomad alloc exec', 'nomad alloc logs', or 'nomad alloc fs' against an allocation whose task already exited or whose client agent is stale; targeting a node that just restarted or was drained; version skew where the remote agent does not support the streaming endpoint.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/bbe3f5ee278740d7. Report an issue: GitHub.