{"record":{"id":"f302ad306b8a4571","repo":"hashicorp/nomad","slug":"unknown-node-q-f302ad","errorCode":null,"errorMessage":"Unknown node %q","messagePattern":"Unknown node %q","errorType":"http","errorClass":null,"httpStatus":400,"severity":"error","filePath":"nomad/client_alloc_endpoint.go","lineNumber":586,"sourceCode":"\t\tif args.JobID != alloc.JobID {\n\t\t\thandleStreamResultError(\n\t\t\t\tfmt.Errorf(\"job %s does not have allocation %s\", args.JobID, alloc.ID),\n\t\t\t\tnew(int64(http.StatusBadRequest)), encoder,\n\t\t\t)\n\t\t}\n\t}\n\n\tnodeID := alloc.NodeID\n\n\t// Make sure Node is valid and new enough to support RPC\n\tnode, err := snap.NodeByID(nil, nodeID)\n\tif err != nil {\n\t\thandleStreamResultError(err, new(int64(500)), encoder)\n\t\treturn\n\t}\n\n\tif node == nil {\n\t\terr := fmt.Errorf(\"Unknown node %q\", nodeID)\n\t\thandleStreamResultError(err, new(int64(400)), encoder)\n\t\treturn\n\t}\n\n\tif err := nodeSupportsRpc(node); err != nil {\n\t\thandleStreamResultError(err, new(int64(400)), encoder)\n\t\treturn\n\t}\n\n\t// Get the connection to the client either by forwarding to another server\n\t// or creating a direct stream\n\tvar clientConn net.Conn\n\tstate, ok := a.srv.getNodeConn(nodeID)\n\tif !ok {\n\t\t// Determine the Server that has a connection to the node.\n\t\tsrv, err := a.srv.serverWithNodeConn(nodeID, a.srv.Region())\n\t\tif err != nil {\n\t\t\tvar code *int64","sourceCodeStart":568,"sourceCodeEnd":604,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/client_alloc_endpoint.go#L568-L604","documentation":"The exec handler resolves the allocation's node (alloc.NodeID) through the state store to obtain a connection to the client. If no node record exists for that ID, the server returns HTTP 400 with \"Unknown node %q\". This means the cluster state no longer contains the node that ran the allocation.","triggerScenarios":"AllocExec targeting an allocation whose NodeID is absent from the Nomad state store (node purged, stale cluster state, or state restored without nodes).","commonSituations":"Node was deregistered/purged while old allocs are still listed; recovering a server from backup snapshots; exec'ing into 'lost' allocations from a decommissioned node.","solutions":["Check the node exists: `nomad node status <node-id>`; if it was purged, the alloc is unreachable — accept and reschedule.","Re-run the job so it schedules onto a live node, then exec into the new allocation.","If nodes are being purged regularly, clean up stale allocs or re-register the node agent."],"exampleFix":"// before\nexec(allocID) // may reference a purged node\n// after\nconst a = getAlloc(allocID)\nif (!nodeExists(a.nodeID)) { rerunJob(a.jobID); return }\nexec(allocID)","handlingStrategy":"validation","validationCode":"const alloc = await nomad.alloc(allocID)\nconst node = await nomad.node(alloc.nodeID).catch(() => null)\nif (!node) {\n  throw new Error(`node ${alloc.nodeID} gone; reschedule before exec`)\n}\nreturn exec(allocID)","typeGuard":null,"tryCatchPattern":"try { await exec(allocID) }\ncatch (e) {\n  if (String(e).includes('Unknown node')) {\n    return rerunJob(getAlloc(allocID).jobID) // alloc is orphaned\n  }\n  throw e\n}","preventionTips":["Don't exec into allocs on nodes recently drained/purged.","Monitor `nomad node status` for deregistered nodes holding stale allocs.","After server state restores, treat old alloc IDs as suspect.","Use garbage collection to clear allocs from unknown nodes."],"tags":["nomad","exec","node","cluster-state"],"backgroundTag":"unknown-node","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}