{"record":{"id":"8034be3d34cb87b3","repo":"hashicorp/nomad","slug":"node-q-not-found","errorCode":null,"errorMessage":"node %q not found","messagePattern":"node %q not found","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/allocwatcher/alloc_watcher.go","lineNumber":509,"sourceCode":"\n\tresp := structs.SingleNodeResponse{}\n\tfor {\n\t\terr := p.rpc.RPC(\"Node.GetNode\", &req, &resp)\n\t\tif err != nil {\n\t\t\tp.logger.Error(\"failed to query node\", \"error\", err, \"node\", nodeID)\n\t\t\tretry := getRemoteRetryIntv + helper.RandomStagger(getRemoteRetryIntv)\n\t\t\tselect {\n\t\t\tcase <-time.After(retry):\n\t\t\t\tcontinue\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn \"\", ctx.Err()\n\t\t\t}\n\t\t}\n\t\tbreak\n\t}\n\n\tif resp.Node == nil {\n\t\treturn \"\", fmt.Errorf(\"node %q not found\", nodeID)\n\t}\n\n\tscheme := \"http://\"\n\tif resp.Node.TLSEnabled {\n\t\tscheme = \"https://\"\n\t}\n\treturn scheme + resp.Node.HTTPAddr, nil\n}\n\n// migrate a remote alloc dir to local node. Caller is responsible for calling\n// Destroy on the returned allocdir if no error occurs.\nfunc (p *remotePrevAlloc) migrateAllocDir(ctx context.Context, nodeAddr string) (*allocdir.AllocDir, error) {\n\t// Create the previous alloc dir\n\tprevAllocDir := allocdir.NewAllocDir(p.logger, p.config.AllocDir, p.config.AllocMountsDir, p.prevAllocID)\n\tif err := prevAllocDir.Build(); err != nil {\n\t\treturn nil, fmt.Errorf(\"error building alloc dir for previous alloc %q: %w\", p.prevAllocID, err)\n\t}\n","sourceCodeStart":491,"sourceCodeEnd":527,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/allocwatcher/alloc_watcher.go#L491-L527","documentation":"In allocwatcher.getNodeAddr, the watcher looks up a node (by nodeID) via the client's state/agent API to obtain the address of the previous allocation's node for migration. If the query response contains a nil Node for the requested nodeID, it returns \"node %q not found\". This means the node referenced by the earlier allocation is unknown to the cluster/state store.","triggerScenarios":"allocWatcher.Migrate (with migrate hooks) calls getNodeAddr for the node ID of the prior alloc; the node lookup (e.g. via the client's CSR/agent API or RPC Nomad.Node-specific query) returns resp.Node == nil - the node ID is not in the state store.","commonSituations":"The previous node was deregistered/garbage-collected (node down past heartbeat, client drained and removed) while an old stopped allocation is being migrated; restoring a cluster from backup/partial state; connecting to the wrong cluster/region so the node ID doesn't resolve; stale allocation references after cluster rebuild.","solutions":["Remove/ignore the stale allocation: purge it (nomad system gc or nomad alloc stop) so migration doesn't reference the dead node","Confirm the client is talking to the correct region/cluster that knows the node ID (check region/datacenter and servers list)","Re-register the node (bring the old client back or let a new client heartbeat) if migration data still lives there","If data locality isn't required, disable migration on the group (migrate { ... } / sticky volumes) so the alloc starts fresh without contacting the old node"],"exampleFix":"// before\nnew alloc tries to migrate from stopped alloc on deregistered node n1 -> error\n// after\ngroup \"web\" {\n  migrate {\n    # or disable sticky/previous-alloc reuse\n  }\n}\n$ nomad system gc","handlingStrategy":"fallback","validationCode":"// Before relying on migration, verify the previous alloc's node is still registered:\nnode, _, err := client.Nodes().Info(prevAlloc.NodeID, nil)\nif err != nil || node == nil {\n    // skip migration; schedule fresh\n}","typeGuard":null,"tryCatchPattern":"// Watcher already falls back; for API consumers:\naddr, err := watcher.getNodeAddr(nodeID)\nif err != nil && strings.Contains(err.Error(), \"not found\") {\n    log.Warnf(\"previous node %q gone; starting alloc without migration\", nodeID)\n    return \"\", nil // proceed without remote migration\n}\nreturn addr, err","preventionTips":["Tune heartbeat TTLs and avoid letting nodes fail GC while sticky allocations reference them","Prefer nomad system gc hygiene after node decommissioning","Use constraints/volumes to avoid sticky migration from decommissioned nodes","Verify region/cluster endpoints so node IDs resolve against the right state store"],"tags":["nomad","alloc-migration","node-lookup","rpc"],"backgroundTag":"node-not-found","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}