hashicorp/nomad · error

Unknown node %q

Error message

Unknown node %q

What it means

The exec handler resolves the allocation's node (alloc.NodeID) through the state store to obtain a connection to the client. If no node record exists for that ID, the server returns HTTP 400 with "Unknown node %q". This means the cluster state no longer contains the node that ran the allocation.

Source

Thrown at nomad/client_alloc_endpoint.go:586

		if args.JobID != alloc.JobID {
			handleStreamResultError(
				fmt.Errorf("job %s does not have allocation %s", args.JobID, alloc.ID),
				new(int64(http.StatusBadRequest)), encoder,
			)
		}
	}

	nodeID := alloc.NodeID

	// Make sure Node is valid and new enough to support RPC
	node, err := snap.NodeByID(nil, nodeID)
	if err != nil {
		handleStreamResultError(err, new(int64(500)), encoder)
		return
	}

	if node == nil {
		err := fmt.Errorf("Unknown node %q", nodeID)
		handleStreamResultError(err, new(int64(400)), encoder)
		return
	}

	if err := nodeSupportsRpc(node); err != nil {
		handleStreamResultError(err, new(int64(400)), encoder)
		return
	}

	// Get the connection to the client either by forwarding to another server
	// or creating a direct stream
	var clientConn net.Conn
	state, ok := a.srv.getNodeConn(nodeID)
	if !ok {
		// Determine the Server that has a connection to the node.
		srv, err := a.srv.serverWithNodeConn(nodeID, a.srv.Region())
		if err != nil {
			var code *int64

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the node exists: `nomad node status <node-id>`; if it was purged, the alloc is unreachable — accept and reschedule.
2. Re-run the job so it schedules onto a live node, then exec into the new allocation.
3. If nodes are being purged regularly, clean up stale allocs or re-register the node agent.

Example fix

// before
exec(allocID) // may reference a purged node
// after
const a = getAlloc(allocID)
if (!nodeExists(a.nodeID)) { rerunJob(a.jobID); return }
exec(allocID)

const alloc = await nomad.alloc(allocID)
const node = await nomad.node(alloc.nodeID).catch(() => null)
if (!node) {
  throw new Error(`node ${alloc.nodeID} gone; reschedule before exec`)
}
return exec(allocID)

try { await exec(allocID) }
catch (e) {
  if (String(e).includes('Unknown node')) {
    return rerunJob(getAlloc(allocID).jobID) // alloc is orphaned
  }
  throw e
}

When it happens

Trigger: AllocExec targeting an allocation whose NodeID is absent from the Nomad state store (node purged, stale cluster state, or state restored without nodes).

Common situations: Node was deregistered/purged while old allocs are still listed; recovering a server from backup snapshots; exec'ing into 'lost' allocations from a decommissioned node.

Related errors

• no exec command is configured
• missing NodeID
• missing NodeID
• Must provide the NodeID
• Invalid node policy: %#v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/f302ad306b8a4571. Report an issue: GitHub.