hashicorp/nomad · error

unknown Nomad server %s

Error message

unknown Nomad server %s

What it means

getServer resolves a server ID to a known peer from the local LAN serf membership. This error means the given serverID matches neither the Name nor ID of any server the node currently knows about — the target server is unknown, has left, or membership data is stale.

Source

Thrown at nomad/rpc.go:741

	return r.srv.connPool.RPC(region, server.Addr, method, args, reply)
}

func (r *rpcHandler) getServer(region, serverID string) (*peers.Parts, error) {

	servers := r.srv.peersCache.RegionPeers(region)
	if len(servers) == 0 {
		r.logger.Warn("no path found to region", "region", region)
		return nil, structs.ErrNoRegionPath
	}

	// Lookup server by id or name
	for _, server := range servers {
		if server.Name == serverID || server.ID == serverID {
			return server, nil
		}
	}

	return nil, fmt.Errorf("unknown Nomad server %s", serverID)
}

// streamingRpc creates a connection to the given server and conducts the
// initial handshake, returning the connection or an error. It is the callers
// responsibility to close the connection if there is no returned error.
func (r *rpcHandler) streamingRpc(server *peers.Parts, method string) (net.Conn, error) {
	c, err := r.srv.connPool.StreamingRPC(r.srv.config.Region, server.Addr)
	if err != nil {
		return nil, err
	}

	return r.streamingRpcImpl(c, method)
}

// streamingRpcImpl takes a pre-established connection to a server and conducts
// the handshake to establish a streaming RPC for the given method. If an error
// is returned, the underlying connection has been closed. Otherwise it is
// assumed that the connection has been hijacked by the RPC method.

View on GitHub (pinned to 482b49bf1a)

Solutions

  1. Verify all servers are healthy members with `nomad server members`
  2. Remove stale/dead members (`nomad server members -prune` or restart gossip healing) and retry
  3. Ensure server data dirs are not reused in a way that changes node IDs unexpectedly
  4. Check network connectivity (serf ports) between servers so membership converges

Example fix

// before
# retrying RPC to stale server id 8f3a... (old server)
// after
nomad server members   # pick a live server
nomad server members -prune  # drop dead members, retry
Defensive patterns

Strategy: retry

Validate before calling

members, _ := client.Agent().Members()
live := false
for _, m := range members.Members {
    if m.Name == serverID || m.ID == serverID { live = true }
}
if !live { return fmt.Errorf("server %s not in LAN membership", serverID) }

Try / catch

srv, err := getServer(servers, serverID)
if err != nil && strings.Contains(err.Error(), "unknown Nomad server") {
    // refresh membership (serf) and retry once
    time.Sleep(time.Second)
    srv, err = getServer(refreshServers(), serverID)
}

Prevention

When it happens

Trigger: Internal RPC paths (e.g. specific-node forwarding, streaming RPC setup) resolving a serverID that is not in serf's LAN member list; peer left/restarted and ID changed; gossip lag after restarts.

Common situations: Server restarted and generated a new node ID while clients cached the old one; split gossip/partitioned LAN; firewall/serf port issues leaving stale member lists; scripts referencing removed servers.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/df427ed3582a8f82. Report an issue: GitHub.