hashicorp/nomad · error

node not found: %s

Error message

node not found: %s

What it means

AgentHostRequest resolves which Nomad node owns the requested nodeID (local client, remote client via forwarding, or server). When the nodeID matches none of those — not this agent's client node, not any server — the handler has no target to RPC 'Agent.Host' to and returns 'node not found: <nodeID>'.

Source

Thrown at command/agent/agent_endpoint.go:873

	lookupNodeID := nodeID
	if serverID != "" {
		lookupNodeID = serverID
	}

	// The RPC endpoint actually forwards the request to the correct
	// agent, but we need to use the correct RPC interface.
	localClient, remoteClient, localServer := s.rpcHandlerForNode(lookupNodeID)
	s.logger.Debug("s.rpcHandlerForNode()", "lookupNodeID", lookupNodeID, "serverID", serverID, "nodeID", nodeID, "localClient", localClient, "remoteClient", remoteClient, "localServer", localServer)

	// Make the RPC call
	if localClient {
		rpcErr = s.agent.Client().ClientRPC("Agent.Host", &args, &reply)
	} else if remoteClient {
		rpcErr = s.agent.Client().RPC("Agent.Host", &args, &reply)
	} else if localServer {
		rpcErr = s.agent.Server().RPC("Agent.Host", &args, &reply)
	} else {
		rpcErr = fmt.Errorf("node not found: %s", nodeID)
	}

	return reply, rpcErr
}

// AgentSchedulerWorkerInfoRequest is used to query the running state of the
// agent's scheduler workers.
func (s *HTTPServer) AgentSchedulerWorkerInfoRequest(resp http.ResponseWriter, req *http.Request) (any, error) {
	srv := s.agent.Server()
	if srv == nil {
		return nil, CodedError(http.StatusBadRequest, ErrServerOnly)
	}
	if req.Method != http.MethodGet {
		return nil, CodedError(http.StatusMethodNotAllowed, ErrInvalidMethod)
	}

	var secret string
	s.parseToken(req, &secret)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Query `nomad node status` and use a current, valid node ID
2. Ensure the request goes to an agent in the same region/datacenter as the node, or to a server that can forward it
3. If targeting the local agent, omit node_id so the handler uses the local client's node
4. If the client was re-provisioned, re-register/look up its fresh node ID

Example fix

// before
curl http://localhost:4646/v1/agent/host?node_id=OLD-STALE-UUID
// after
NODE_ID=$(nomad node status -json | jq -r '.[0].ID')
curl http://localhost:4646/v1/agent/host?node_id=$NODE_ID

nodeID := "..." // from nomad node status
if nodeID == "" || !uuidRegexp.MatchString(nodeID) {
    return errors.New("invalid or missing node_id")
}

resp, err := client.Agent().Host(nodeID, nil)
if err != nil {
    if strings.Contains(err.Error(), "node not found") {
        // refresh node list and retry with a valid ID
        nodes, _ := client.Nodes().List(nil)
        ...
    }
    return err
}

When it happens

Trigger: GET /v1/agent/host?node_id=<id> where nodeID is a UUID that is neither the local Nomad client's node ID nor any server member's ID: stale/deleted node ID, wrong datacenter (no remote client path), or querying against a server-only agent with a client node's ID.

Common situations: Scripting host stats collection with a node ID captured before the client was restarted (new node ID generated); targeting a node in another region without server forwarding; typo'd node ID in an automation tool.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• missing NodeID
• missing NodeID
• Must provide the NodeID
• Invalid node policy: %#v
• http addr of node %q (%s) is not advertised

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/2d27dd152b624fe3. Report an issue: GitHub.