{"record":{"id":"11f05e55db99b869","repo":"hashicorp/nomad","slug":"failed-querying-server-q-v","errorCode":null,"errorMessage":"failed querying server %q: %v","messagePattern":"failed querying server %q: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/client_rpc.go","lineNumber":184,"sourceCode":"\t\t}\n\t}\n\n\t// connections is used to store the servers that have connections to the\n\t// requested node.\n\tvar mostRecentServer *peers.Parts\n\tvar mostRecent time.Time\n\n\tvar rpcErr multierror.Error\n\tfor _, server := range targets {\n\t\tif server.Addr.String() == selfAddr {\n\t\t\tcontinue\n\t\t}\n\n\t\t// Make the RPC\n\t\tvar resp structs.NodeConnQueryResponse\n\t\terr := s.connPool.RPC(s.config.Region, server.Addr, \"Status.HasNodeConn\", &req, &resp)\n\t\tif err != nil {\n\t\t\tmultierror.Append(&rpcErr, fmt.Errorf(\"failed querying server %q: %v\", server.Addr.String(), err))\n\t\t\tcontinue\n\t\t}\n\n\t\tif resp.Connected && resp.Established.After(mostRecent) {\n\t\t\tmostRecentServer = server\n\t\t\tmostRecent = resp.Established\n\t\t}\n\t}\n\n\t// Return an error if there is no route to the node.\n\tif mostRecentServer == nil {\n\t\tif err := rpcErr.ErrorOrNil(); err != nil {\n\t\t\treturn nil, err\n\t\t}\n\n\t\treturn nil, structs.ErrNoNodeConn\n\t}\n","sourceCodeStart":166,"sourceCodeEnd":202,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/client_rpc.go#L166-L202","documentation":"serverWithNodeConn iterates servers asking each via Status.HasNodeConn whether it holds a multiplexed connection to the target node; failures querying an individual server are accumulated into rpcErr with this message including the server address. It signals a per-server RPC failure while searching for the best server to forward a client-targeted streaming/exec RPC.","triggerScenarios":"Called from findClientConn / exec / streaming fs handlers / findNodeConnAndForward when the connPool.RPC to Status.HasNodeConn on some server fails: server down, TLS/mTLS misconfiguration, wrong region advertise address, network partition, or RPC timeout.","commonSituations":"A peer server crashed or is being replaced; TLS certificate mismatch between servers; firewall blocking RPC port (default 4647) between servers; servers in different regions/datacenters with broken WAN gossip.","solutions":["Check the wrapped %v for the actual cause (timeout, TLS handshake, connection refused).","Verify all servers are healthy: `nomad server members` and `nomad server peers`.","Confirm server-to-server connectivity on the RPC port (4647) and matching TLS CA/certificates.","If one specific server is bad (shown in %q), restart or remove it from the Raft peers."],"exampleFix":"// before\nserver 10.0.0.5:4647 unreachable (mTLS)\n// after\nnomad agent -server -ca-file=ca.pem -cert-file=server.pem -key-file=server-key.pem ...  # matching certs on all servers","handlingStrategy":"retry","validationCode":"// health-check peers first\nfor _, m := range client.Agent().Members().ServerMembers {\n    if m.Status != \"alive\" { log.Printf(\"server %s not alive\", m.Name) }\n}","typeGuard":null,"tryCatchPattern":"if strings.Contains(err.Error(), \"failed querying server\") {\n    // transient per-server failure; retry — iteration covers other peers\n    time.Sleep(2 * time.Second)\n    return findNodeConnAndForward(...) // retry\n}","preventionTips":["Keep all servers alive and on matching TLS configs.","Ensure server RPC port 4647 is open between all server nodes.","Monitor `nomad server members` for failed/left peers."],"tags":["rpc","networking","nomad","server-forwarding"],"backgroundTag":"server-rpc-unreachable","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}