{"record":{"id":"bbe3f5ee278740d7","repo":"hashicorp/nomad","slug":"remote-node-streaming-rpc-error-relayed-from-ack","errorCode":null,"errorMessage":"<remote node streaming RPC error relayed from ack.Error>","messagePattern":"<remote node streaming RPC error relayed from ack\\.Error>","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/client_rpc.go","lineNumber":302,"sourceCode":"\tdecoder := codec.NewDecoder(stream, structs.MsgpackHandle)\n\theader := structs.StreamingRpcHeader{\n\t\tMethod: method,\n\t}\n\tif err := encoder.Encode(header); err != nil {\n\t\tstream.Close()\n\t\treturn nil, err\n\t}\n\n\t// Wait for the acknowledgement\n\tvar ack structs.StreamingRpcAck\n\tif err := decoder.Decode(&ack); err != nil {\n\t\tstream.Close()\n\t\treturn nil, err\n\t}\n\n\tif ack.Error != \"\" {\n\t\tstream.Close()\n\t\treturn nil, errors.New(ack.Error)\n\t}\n\n\treturn stream, nil\n}\n\n// findNodeConnAndForward is a helper for finding the server with a connection\n// to the given node and forwarding the RPC to the correct server. This does not\n// work for streaming RPCs.\nfunc findNodeConnAndForward(srv *Server, nodeID, method string, args, reply any) error {\n\t// Determine the Server that has a connection to the node.\n\tsrvWithConn, err := srv.serverWithNodeConn(nodeID, srv.Region())\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tif srvWithConn == nil {\n\t\treturn structs.ErrNoNodeConn\n\t}","sourceCodeStart":284,"sourceCodeEnd":320,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/client_rpc.go#L284-L320","documentation":"NodeStreamingRpc on the Nomad client forwards a streaming RPC to the remote node's agent. When the remote node replies with a Nack-style ack containing a non-empty Error field, the client closes the stream and relays that message verbatim as a new error via errors.New(ack.Error). The '<remote node streaming RPC error relayed from ack.Error>' prefix indicates the real failure happened on another node, not the server handling the request.","triggerScenarios":"A server calls NodeStreamingRpc to reach a client node (used by exec, stream, logs, and forwardMonitorClient paths); the target node's agent accepts the connection but responds with an ack whose Error field is set, e.g. the exec task is gone, the file does not exist, or the node-side handler rejected the stream.","commonSituations":"Running 'nomad alloc exec', 'nomad alloc logs', or 'nomad alloc fs' against an allocation whose task already exited or whose client agent is stale; targeting a node that just restarted or was drained; version skew where the remote agent does not support the streaming endpoint.","solutions":["Run 'nomad node status <node>' and check the target allocation/task state; retry after confirming the task is running","Verify the client agent on the target node is running and matches the server version (upgrade to align)","Re-run the command against the correct allocation ID if the original task finished","Check the target node's nomad agent logs for the underlying handler error that was relayed"],"exampleFix":"// before (command against stale alloc)\nnomad alloc exec -task web 1a2b3c ls /\n// error: remote node relayed failure\n// after: confirm task is running first\nnomad alloc status 1a2b3c   # ensure task 'web' is running\nnomad alloc exec -task web <current-alloc-id> ls /","handlingStrategy":"retry","validationCode":"// shell: verify allocation and task are running before streaming\nnomad alloc status <allocID> | grep 'Task .* running' || { echo 'task not running'; exit 1; }","typeGuard":null,"tryCatchPattern":"// Go: retry the streaming RPC on relayed errors\nvar stream *structs.ClientRPCStream\nerr := retry.Do(func() error {\n    s, err := node.NodeStreamingRpc(nodeID, req)\n    if err != nil {\n        if strings.Contains(err.Error(), \"relayed from ack.Error\") {\n            return retry.TransientError{Err: err} // remote node issue; retry\n        }\n        return retry.Unrecoverable(err)\n    }\n    stream = s\n    return nil\n})","preventionTips":["Check alloc/task health before exec/logs/fs commands","Keep client and server agents on compatible versions","Monitor node liveness and drained/stale states before issuing node RPCs","Read the relayed message — it names the node-side cause; check that node's agent logs"],"tags":["nomad","rpc","streaming","remote-node"],"backgroundTag":"remote-rpc-error-relayed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}