hashicorp/nomad · error

Unknown node %q

Error message

Unknown node %q

What it means

findClientConn resolves the client RPC connection for a given node ID by looking the node up in the state store. If the node ID is unknown, it wraps this error in a 404 ErrRPCCoded response. It indicates the requested client node does not exist in the server's state.

Source

Thrown at nomad/client_agent_endpoint.go:583

	return nil
}

// findClientConn is a helper that returns a connection to the client node or, if the client
// is connected to a different server, a serverParts describing the server to which the
// client bound RPC should be forwarded.
func (a *Agent) findClientConn(nodeID string) (*nodeConnState, *peers.Parts, error) {
	snap, err := a.srv.State().Snapshot()
	if err != nil {
		return nil, nil, structs.NewErrRPCCoded(500, err.Error())
	}

	node, err := snap.NodeByID(nil, nodeID)
	if err != nil {
		return nil, nil, structs.NewErrRPCCoded(500, err.Error())
	}

	if node == nil {
		err := fmt.Errorf("Unknown node %q", nodeID)
		return nil, nil, structs.NewErrRPCCoded(404, err.Error())
	}

	if err := nodeSupportsRpc(node); err != nil {
		return nil, nil, structs.NewErrRPCCoded(400, err.Error())
	}

	// Get the Connection to the client either by fowarding to another server
	// or creating direct stream
	state, ok := a.srv.getNodeConn(nodeID)
	if ok {
		return state, nil, nil
	}

	// Determine the server that has a connection to the node
	srv, err := a.srv.serverWithNodeConn(nodeID, a.srv.Region())
	if err != nil {
		code := 500

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify the node ID with `nomad node status` and use the exact ID of a registered client.
2. Re-register the client node (restart the nomad agent on the client) if it was GC'd.
3. Ensure you are querying the correct region where the node is registered.

Example fix

// before
conn, _, err := findClientConn(ctx, args, "missing-node-id") // 404 Unknown node
// after
nodes, _ := client.Nodes().List(nil)
nodeID := pickExistingNode(nodes) // use a Node.ID from node status

node, _, err := client.Nodes().Info(nodeID, nil)
if err != nil || node == nil {
    return fmt.Errorf("node %s not registered; refresh node list", nodeID)
}

conn, _, err := findClientConn(ctx, args, nodeID)
var rpcErr *structs.RPCError
if errors.As(err, &rpcErr) && rpcErr.Code == 404 {
    // node unknown: refresh node status and re-select a valid node
}

When it happens

Trigger: Calling forwardMonitorClient, forwardProfileClient, or Host targeting a client node whose NodeID is absent from the state store (snap.NodeByID returns nil).

Common situations: Node was deregistered/GC'd after inactivity; typo'd node ID; targeting a node in a different region/cluster; stale UI sessions referencing removed clients; running node against a cluster it never joined.

Related errors

• missing NodeID
• missing NodeID
• Must provide the NodeID
• Unknown node
• No node(s) with prefix or id %q found

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/d8c4eb4846b5793a. Report an issue: GitHub.