{"record":{"id":"c503587c4ae52f1b","repo":"hashicorp/nomad","slug":"set-mode-v","errorCode":null,"errorMessage":"set mode: %v","messagePattern":"set mode: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/client_rpc.go","lineNumber":253,"sourceCode":"\n\t// Make the RPC\n\treturn NodeRpc(state.Session, method, args, reply)\n}\n\n// NodeRpc is used to make an RPC call to a node. The method takes the\n// Yamux session for the node and the method to be called.\nfunc NodeRpc(session *yamux.Session, method string, args, reply any) error {\n\t// Open a new session\n\tstream, err := session.Open()\n\tif err != nil {\n\t\treturn fmt.Errorf(\"session open: %v\", err)\n\t}\n\tdefer stream.Close()\n\n\t// Write the RpcNomad byte to set the mode\n\tif _, err := stream.Write([]byte{byte(pool.RpcNomad)}); err != nil {\n\t\tstream.Close()\n\t\treturn fmt.Errorf(\"set mode: %v\", err)\n\t}\n\n\t// Make the RPC\n\terr = msgpackrpc.CallWithCodec(pool.NewClientCodec(stream), method, args, reply)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\treturn nil\n}\n\n// NodeStreamingRpc is used to make a streaming RPC call to a node. The method\n// takes the Yamux session for the node and the method to be called. It conducts\n// the initial handshake and returns a connection to be used or an error. It is\n// the callers responsibility to close the connection if there is no error.\nfunc NodeStreamingRpc(session *yamux.Session, method string) (net.Conn, error) {\n\t// Open a new session\n\tstream, err := session.Open()","sourceCodeStart":235,"sourceCodeEnd":271,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/client_rpc.go#L235-L271","documentation":"NodeRpc writes a single mode byte (pool.RpcNomad) after opening the stream so the client's mux reader dispatches to the Nomad RPC handler; failure writing that byte produces this error. It means the stream broke immediately after opening — the connection to the node is effectively unusable.","triggerScenarios":"Any NodeRpc caller (Signal, exec/Host streaming, pause-state, GC, profiling) where the stream.Write of the RpcNomad byte fails: connection reset, client disconnected concurrently, or stream closed by peer.","commonSituations":"Client agent stopping at the exact moment of the call; flaky network dropping the muxed TCP connection; container/node shutdown racing with an admin command like `nomad alloc signal`.","solutions":["Retry the command; a fresh connection will be dialed if the old one is dead.","Verify node health: `nomad node status <node-id>` should show ready.","Check the client's logs for connection resets and fix underlying network instability.","If the alloc's client is being shut down, wait for rescheduling and target the new alloc."],"exampleFix":"// before\nnomad alloc signal -task web SIGTERM <alloc>  # fails: set mode: connection reset\n// after (retry once after checking node)\nnomad node status <node> && nomad alloc signal -task web SIGTERM <alloc>","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"if strings.Contains(err.Error(), \"set mode\") {\n    // stream died immediately; reconnect and retry once\n    time.Sleep(500 * time.Millisecond)\n    return NodeRpc(session, method, args, reply)\n}","preventionTips":["Retry idempotent node RPCs (signal, pause state) on transient stream failures.","Stagger commands during rolling client restarts.","Monitor client agent uptime and connection churn."],"tags":["rpc","yamux","streaming","nomad"],"backgroundTag":"rpc-stream-write-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}