{"record":{"id":"f1fe0dc772a557e7","repo":"hashicorp/nomad","slug":"session-open-v","errorCode":null,"errorMessage":"session open: %v","messagePattern":"session open: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/client_rpc.go","lineNumber":246,"sourceCode":"\n\t// Get the connection to the client\n\tstate, ok := s.getNodeConn(nodeID)\n\tif !ok {\n\t\t// Make the RPC via another server\n\t\treturn findNodeConnAndForward(s, nodeID, method, args, reply)\n\t}\n\n\t// Make the RPC\n\treturn NodeRpc(state.Session, method, args, reply)\n}\n\n// NodeRpc is used to make an RPC call to a node. The method takes the\n// Yamux session for the node and the method to be called.\nfunc NodeRpc(session *yamux.Session, method string, args, reply any) error {\n\t// Open a new session\n\tstream, err := session.Open()\n\tif err != nil {\n\t\treturn fmt.Errorf(\"session open: %v\", err)\n\t}\n\tdefer stream.Close()\n\n\t// Write the RpcNomad byte to set the mode\n\tif _, err := stream.Write([]byte{byte(pool.RpcNomad)}); err != nil {\n\t\tstream.Close()\n\t\treturn fmt.Errorf(\"set mode: %v\", err)\n\t}\n\n\t// Make the RPC\n\terr = msgpackrpc.CallWithCodec(pool.NewClientCodec(stream), method, args, reply)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\treturn nil\n}\n","sourceCodeStart":228,"sourceCodeEnd":264,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/client_rpc.go#L228-L264","documentation":"NodeRpc opens a new yamux stream over the multiplexed session to a node; this error is returned when session.Open() fails, meaning the underlying connection or session to the client cannot accept a new stream. It usually indicates the whole node connection is dead or closing.","triggerScenarios":"Called by forwardProfileClient, Host (fs/exec endpoints), GarbageCollectAll, Signal, SetPauseState, GetPauseState when the yamux session to the node is already closed, the TCP connection dropped, or the session exhausted streams.","commonSituations":"Client agent restarted or crashed between connection check and RPC; NAT/firewall dropped the idle multiplexed connection; network blip; client in the middle of shutdown while a task command (signal, exec, logs) is issued.","solutions":["Retry the operation; the server will typically re-establish the node connection.","Check the client agent is running and reachable: `nomad node status <node>` shows ready.","Inspect network devices/firewalls for idle timeouts killing long-lived mux connections; raise idle timeouts or enable keepalives.","If it persists, restart the client agent to reset its RPC multiplexing."],"exampleFix":"// before: single attempt\nerr := NodeRpc(session, \"Exec\", args, reply)\n// after: retry on session failure\nvar err error\nfor i := 0; i < 3; i++ {\n    if err = NodeRpc(session, \"Exec\", args, reply); err == nil || !strings.Contains(err.Error(), \"session open\") {\n        break\n    }\n    time.Sleep(time.Second)\n}","handlingStrategy":"retry","validationCode":"node, _, err := client.Nodes().Info(nodeID, nil)\nif err != nil || node.Status != \"ready\" {\n    return fmt.Errorf(\"skip NodeRpc: node %s not ready\", nodeID)\n}","typeGuard":null,"tryCatchPattern":"if strings.Contains(err.Error(), \"session open\") {\n    // dead mux session; brief backoff then retry once\n    time.Sleep(time.Second)\n    return NodeRpc(session, method, args, reply)\n}","preventionTips":["Verify node readiness before signals/exec/log RPCs.","Tune firewall/NAT idle timeouts above Nomad's mux keepalive interval.","Avoid RPC bursts to clients mid-restart; use health checks."],"tags":["rpc","yamux","networking","nomad"],"backgroundTag":"session-open-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}