What causes "unknown nomad server %s" in hashicorp/nomad?

List valid targets with `nomad server members` (or Members() API) and use the exact serf Name of a live server.

unknown nomad server %s

Error message

unknown nomad server %s

What it means

forwardFor locates a target server by server ID among the local region's member list. If no server with the given ID exists in the LAN Serf membership, target stays nil and this error is returned. It means the requested server name doesn't match any known Nomad server.

Source

Thrown at nomad/client_agent_endpoint.go:413

		isLeader, remoteLeader := a.srv.getLeader()
		if !isLeader && remoteLeader != nil {
			target = remoteLeader
		} else if !isLeader && remoteLeader == nil {
			return nil, structs.ErrNoLeader
		} else if isLeader {
			// This server is current leader do not forward
			return nil, nil
		}
	} else {
		target, err = a.srv.getServer(region, serverID)
		if err != nil {
			return nil, err
		}
	}

	// Unable to find a server
	if target == nil {
		return nil, fmt.Errorf("unknown nomad server %s", serverID)
	}

	// ServerID is this current server,
	// No need to forward request
	if target.Name == a.srv.LocalMember().Name {
		return nil, nil
	}

	return target, nil
}

func (a *Agent) forwardMonitorClient(conn io.ReadWriteCloser, args any, encoder *codec.Encoder, decoder *codec.Decoder, nodeID string, endpoint string) {
	// Get the Connection to the client either by fowarding to another server
	// or creating direct stream
	state, srv, err := a.findClientConn(nodeID)
	if err != nil {
		handleStreamResultError(err, new(int64(500)), encoder)
		return

View on GitHub (pinned to 482b49bf1a)

Solutions

List valid targets with `nomad server members` (or Members() API) and use the exact serf Name of a live server.
Retry after new servers join; a freshly started server may not yet be visible.
If the server was replaced, target the new server's name instead of the old ID.

Example fix

// before
reply, err := endpoint.Profile(args) // args.ServerID = "old-server" (decommissioned)
// after
members, _ := client.Agent().Members()
args.ServerID = pickLiveServer(members) // e.g. "ip-10-0-0-5.node.dc1"

Defensive patterns

Strategy: validation

Validate before calling

members, err := client.Agent().Members()
if err != nil { return err }
valid := false
for _, m := range members.Members {
    if m.Name == serverID { valid = true; break }
}
if !valid { return fmt.Errorf("server %q not in member list", serverID) }

Try / catch

target, err := forwardFor(ctx, args, serverID)
if err != nil && strings.HasPrefix(err.Error(), "unknown nomad server") {
    // refresh member list and retry with a live server name
}

Prevention

Select server targets from `nomad server members`, not hostnames.
Re-resolve server names after upgrades/decommissions; serf names change on replacement.
Refresh the member list before each targeted agent RPC.

When it happens

Trigger: Calling Profile, monitor, monitorExport, or Host with a ServerID/ServerName parameter that does not match any member returned by the local memberlist — the exact case in forwardFor when target == nil after scanning members.

Common situations: Typo in server name when selecting a node for profiling/logs; targeting a server that was decommissioned or replaced (names regenerate); querying a region whose member list doesn't include the target; DNS vs serf name confusion (serf name is not the hostname).

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/b4f069318ecbe684. Report an issue: GitHub.