{"record":{"id":"df427ed3582a8f82","repo":"hashicorp/nomad","slug":"unknown-nomad-server-s-df427e","errorCode":null,"errorMessage":"unknown Nomad server %s","messagePattern":"unknown Nomad server (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/rpc.go","lineNumber":741,"sourceCode":"\treturn r.srv.connPool.RPC(region, server.Addr, method, args, reply)\n}\n\nfunc (r *rpcHandler) getServer(region, serverID string) (*peers.Parts, error) {\n\n\tservers := r.srv.peersCache.RegionPeers(region)\n\tif len(servers) == 0 {\n\t\tr.logger.Warn(\"no path found to region\", \"region\", region)\n\t\treturn nil, structs.ErrNoRegionPath\n\t}\n\n\t// Lookup server by id or name\n\tfor _, server := range servers {\n\t\tif server.Name == serverID || server.ID == serverID {\n\t\t\treturn server, nil\n\t\t}\n\t}\n\n\treturn nil, fmt.Errorf(\"unknown Nomad server %s\", serverID)\n}\n\n// streamingRpc creates a connection to the given server and conducts the\n// initial handshake, returning the connection or an error. It is the callers\n// responsibility to close the connection if there is no returned error.\nfunc (r *rpcHandler) streamingRpc(server *peers.Parts, method string) (net.Conn, error) {\n\tc, err := r.srv.connPool.StreamingRPC(r.srv.config.Region, server.Addr)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\treturn r.streamingRpcImpl(c, method)\n}\n\n// streamingRpcImpl takes a pre-established connection to a server and conducts\n// the handshake to establish a streaming RPC for the given method. If an error\n// is returned, the underlying connection has been closed. Otherwise it is\n// assumed that the connection has been hijacked by the RPC method.","sourceCodeStart":723,"sourceCodeEnd":759,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/rpc.go#L723-L759","documentation":"getServer resolves a server ID to a known peer from the local LAN serf membership. This error means the given serverID matches neither the Name nor ID of any server the node currently knows about — the target server is unknown, has left, or membership data is stale.","triggerScenarios":"Internal RPC paths (e.g. specific-node forwarding, streaming RPC setup) resolving a serverID that is not in serf's LAN member list; peer left/restarted and ID changed; gossip lag after restarts.","commonSituations":"Server restarted and generated a new node ID while clients cached the old one; split gossip/partitioned LAN; firewall/serf port issues leaving stale member lists; scripts referencing removed servers.","solutions":["Verify all servers are healthy members with `nomad server members`","Remove stale/dead members (`nomad server members -prune` or restart gossip healing) and retry","Ensure server data dirs are not reused in a way that changes node IDs unexpectedly","Check network connectivity (serf ports) between servers so membership converges"],"exampleFix":"// before\n# retrying RPC to stale server id 8f3a... (old server)\n// after\nnomad server members   # pick a live server\nnomad server members -prune  # drop dead members, retry","handlingStrategy":"retry","validationCode":"members, _ := client.Agent().Members()\nlive := false\nfor _, m := range members.Members {\n    if m.Name == serverID || m.ID == serverID { live = true }\n}\nif !live { return fmt.Errorf(\"server %s not in LAN membership\", serverID) }","typeGuard":null,"tryCatchPattern":"srv, err := getServer(servers, serverID)\nif err != nil && strings.Contains(err.Error(), \"unknown Nomad server\") {\n    // refresh membership (serf) and retry once\n    time.Sleep(time.Second)\n    srv, err = getServer(refreshServers(), serverID)\n}","preventionTips":["Monitor serf LAN membership health","Prune dead members after server decommissions","Avoid caching server IDs across restarts; resolve via membership each time"],"tags":["nomad","rpc","serf","membership","cluster"],"backgroundTag":"unknown-server-id","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}