{"record":{"id":"2c3702d02a6ce51a","repo":"hashicorp/nomad","slug":"failed-to-find-allocs-for-s-v","errorCode":null,"errorMessage":"failed to find allocs for '%s': %v","messagePattern":"failed to find allocs for '(.+?)': (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"nomad/node_endpoint.go","lineNumber":1819,"sourceCode":"\t\t}}\n\treturn n.srv.blockingRPC(&opts)\n}\n\n// createNodeEvals is used to create evaluations for each alloc on a node.\n// Each Eval is scoped to a job, so we need to potentially trigger many evals.\nfunc (n *Node) createNodeEvals(node *structs.Node, nodeIndex uint64) ([]string, uint64, error) {\n\tnodeID := node.ID\n\n\t// Snapshot the state\n\tsnap, err := n.srv.fsm.State().Snapshot()\n\tif err != nil {\n\t\treturn nil, 0, fmt.Errorf(\"failed to snapshot state: %v\", err)\n\t}\n\n\t// Find all the allocations for this node\n\tallocs, err := snap.AllocsByNode(nil, nodeID)\n\tif err != nil {\n\t\treturn nil, 0, fmt.Errorf(\"failed to find allocs for '%s': %v\", nodeID, err)\n\t}\n\n\tsysJobsIter, err := snap.JobsByScheduler(nil, \"system\")\n\tif err != nil {\n\t\treturn nil, 0, fmt.Errorf(\"failed to find system jobs for '%s': %v\", nodeID, err)\n\t}\n\n\tvar sysJobs []*structs.Job\n\tfor jobI := sysJobsIter.Next(); jobI != nil; jobI = sysJobsIter.Next() {\n\t\tjob := jobI.(*structs.Job)\n\t\t// Avoid creating evals for jobs that don't run in this datacenter or\n\t\t// node pool. We could perform an entire feasibility check here, but\n\t\t// datacenter/pool is a good optimization to start with as their\n\t\t// cardinality tends to be low so the check shouldn't add much work.\n\t\t// If the job is stopped, skip it as well, otherwise we will create an\n\t\t// eval with state and broker overhead that will be an immediate no-op.\n\t\tif node.IsInPool(job.NodePool) && node.IsInAnyDC(job.Datacenters) && !job.Stopped() {\n\t\t\tsysJobs = append(sysJobs, job)","sourceCodeStart":1801,"sourceCodeEnd":1837,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/node_endpoint.go#L1801-L1837","documentation":"After snapshotting state, createNodeEvals looks up all allocations on the node via snap.AllocsByNode. If that state-store read fails, the error is wrapped with the node ID. Like the snapshot failure, it signals an internal state-store read error rather than invalid user input.","triggerScenarios":"Register, deregister, UpdateStatus, UpdateDrain, UpdateEligibility, or Evaluate RPCs for a node whose allocation index lookup in the state store snapshot fails (internal memdb/iterator error).","commonSituations":"State store iterator failures under memory pressure, corrupted state store, or extremely large allocation tables causing resource exhaustion during iteration.","solutions":["Restart the affected Nomad server to rebuild a clean in-memory state store","Inspect server logs around the error for the underlying memdb message","Check host memory/CPU; a very large cluster may need more server capacity","Restore from backup or re-add the node if the state store is corrupt"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"resp, err := client.Nodes().Register(req)\nif err != nil && strings.Contains(err.Error(), \"failed to find allocs for\") {\n    // retry with backoff; if persistent, restart the server\n}","preventionTips":["Watch for cluster size outgrowing server capacity","Restart unhealthy servers promptly","Keep Nomad servers patched to latest stable","Monitor memdb/state-store errors in logs"],"tags":["nomad","state-store","allocations","internal"],"backgroundTag":"state-store-corruption","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}