{"record":{"id":"617b33e90990922f","repo":"hashicorp/nomad","slug":"rpc-error-w","errorCode":null,"errorMessage":"rpc error: %w","messagePattern":"rpc error: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"helper/pool/pool.go","lineNumber":517,"sourceCode":"\ts, err := conn.session.Open()\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to open a streaming connection: %v\", err)\n\t}\n\n\tif _, err := s.Write([]byte{byte(RpcStreaming)}); err != nil {\n\t\tconn.Close()\n\t\treturn nil, err\n\t}\n\n\treturn s, nil\n}\n\n// RPC is used to make an RPC call to a remote host\nfunc (p *ConnPool) RPC(region string, addr net.Addr, method string, args interface{}, reply interface{}) error {\n\t// Get a usable client\n\tconn, sc, err := p.getRPCClient(region, addr)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"rpc error: %w\", err)\n\t}\n\tdefer conn.releaseUse()\n\n\t// Make the RPC call\n\terr = msgpackrpc.CallWithCodec(sc.codec, method, args, reply)\n\tif err != nil {\n\t\tsc.Close()\n\n\t\t// If we read EOF, the session is toast. Clear it and open a\n\t\t// new session next time\n\t\t// See https://github.com/hashicorp/consul/blob/v1.6.3/agent/pool/pool.go#L471-L477\n\t\tif helper.IsErrEOF(err) {\n\t\t\tp.clearConn(conn)\n\t\t}\n\n\t\t// If the error is an RPC Coded error\n\t\t// return the coded error without wrapping\n\t\tif structs.IsErrRPCCoded(err) {","sourceCodeStart":499,"sourceCodeEnd":535,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/helper/pool/pool.go#L499-L535","documentation":"ConnPool.RPC fails to establish an RPC client connection to a remote Nomad server and wraps the underlying error with \"rpc error: %w\". This is a transport-level failure in the connection pool's getRPCClient step (dial, handshake, or pool acquisition), not an application-level RPC error. The wrapped error (%w) contains the root cause.","triggerScenarios":"Calling ConnPool.RPC (or Ping, serverWithNodeConn, maybeBootstrap, fetch which all route through RPC) when getRPCClient cannot obtain a usable client: target host unreachable, wrong port, TLS/mTLS misconfiguration, or pool connection establishment timeout.","commonSituations":"Server down or restarting during rolling upgrades; firewall blocking the RPC port (default 4647); advertise address misconfigured so peers dial a wrong IP; TLS certificate mismatch between client and server; DNS resolution failure in the region.","solutions":["Unwrap the error (%s of the cause) to identify whether it is dial failure, TLS, or timeout, and fix that root cause first","Verify the target server is up and listening on the RPC port (nomad server-members / netstat on 4647)","Check advertise_rpc / advertise addresses and firewall rules allow RPC between nodes","If TLS is enabled, verify ca_file, cert_file, key_file and verify_server_hostname consistency across the cluster"],"exampleFix":"// before\nerr := pool.RPC(region, addr, \"Status.Ping\", args, reply)\nlog.Printf(\"ping failed: %v\", err)\n// after\nif err := pool.RPC(region, addr, \"Status.Ping\", args, reply); err != nil {\n    var nerr net.Error\n    if errors.As(err, &nerr) && nerr.Timeout() {\n        log.Printf(\"RPC timeout to %s, retrying: %v\", addr, err)\n    } else {\n        log.Printf(\"RPC transport failure to %s: %v\", addr, err)\n    }\n}","handlingStrategy":"retry","validationCode":"conn, err := net.DialTimeout(\"tcp\", addr.String(), 3*time.Second)\nif err != nil {\n    return fmt.Errorf(\"RPC endpoint %s unreachable before call: %w\", addr, err)\n}\nconn.Close()","typeGuard":null,"tryCatchPattern":"err := pool.RPC(region, addr, method, args, reply)\nvar nerr net.Error\nif errors.As(err, &nerr) && (nerr.Timeout() || isTransient(nerr)) {\n    return retryWithBackoff(call)\n}\nreturn err","preventionTips":["Health-check the endpoint (Ping) before issuing application RPCs","Keep advertise/addresses consistent and firewall RPC port 4647 open between nodes","Validate TLS configuration across the cluster before rollout","Use retry with backoff for transient transport errors"],"tags":["network","rpc","connection","go"],"backgroundTag":"rpc-connection-failure","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}