{"record":{"id":"215b226d2ff268b3","repo":"hashicorp/nomad","slug":"failed-to-snapshot-state-v","errorCode":null,"errorMessage":"failed to snapshot state: %v","messagePattern":"failed to snapshot state: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"nomad/node_endpoint.go","lineNumber":1813,"sourceCode":"\t\t\t}\n\t\t\treply.Index = index\n\n\t\t\t// Set the query response\n\t\t\tn.srv.setQueryMeta(&reply.QueryMeta)\n\t\t\treturn nil\n\t\t}}\n\treturn n.srv.blockingRPC(&opts)\n}\n\n// createNodeEvals is used to create evaluations for each alloc on a node.\n// Each Eval is scoped to a job, so we need to potentially trigger many evals.\nfunc (n *Node) createNodeEvals(node *structs.Node, nodeIndex uint64) ([]string, uint64, error) {\n\tnodeID := node.ID\n\n\t// Snapshot the state\n\tsnap, err := n.srv.fsm.State().Snapshot()\n\tif err != nil {\n\t\treturn nil, 0, fmt.Errorf(\"failed to snapshot state: %v\", err)\n\t}\n\n\t// Find all the allocations for this node\n\tallocs, err := snap.AllocsByNode(nil, nodeID)\n\tif err != nil {\n\t\treturn nil, 0, fmt.Errorf(\"failed to find allocs for '%s': %v\", nodeID, err)\n\t}\n\n\tsysJobsIter, err := snap.JobsByScheduler(nil, \"system\")\n\tif err != nil {\n\t\treturn nil, 0, fmt.Errorf(\"failed to find system jobs for '%s': %v\", nodeID, err)\n\t}\n\n\tvar sysJobs []*structs.Job\n\tfor jobI := sysJobsIter.Next(); jobI != nil; jobI = sysJobsIter.Next() {\n\t\tjob := jobI.(*structs.Job)\n\t\t// Avoid creating evals for jobs that don't run in this datacenter or\n\t\t// node pool. We could perform an entire feasibility check here, but","sourceCodeStart":1795,"sourceCodeEnd":1831,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/node_endpoint.go#L1795-L1831","documentation":"createNodeEvals snapshots the Raft-backed state store before scanning allocations. If the in-memory state store cannot produce a consistent snapshot (an internal memdb error), the endpoint returns this wrapped error. It indicates internal state-store trouble, not a user-input problem.","triggerScenarios":"Any RPC that leads to eval creation — Register, deregister, UpdateStatus, UpdateDrain, UpdateEligibility, Evaluate — when n.srv.fsm.State().Snapshot() returns an error (state store corruption, memory pressure, internal memdb failure).","commonSituations":"Server under severe memory pressure, corrupted state store after a crash, or bugs in the FSM state store; often appears alongside other state-store errors on the same server.","solutions":["Restart the affected Nomad server (followers first, then leader via a controlled leader election)","Check server logs for preceding state-store or memdb errors to identify the root cause","Verify memory resources on the server host; increase if OOM/thrashing is suspected","If corruption persists, restore the server from a backup or rebuild the node by re-adding it"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// server-side: check state-store health before heavy RPCs\nif err := fsm.State().Snapshot(); err != nil { /* backoff and retry RPC */ }","typeGuard":null,"tryCatchPattern":"var evalErr *evals.CreateFailedEvalError // pseudo\nresp, err := client.Nodes().UpdateStatus(req)\nif err != nil && strings.Contains(err.Error(), \"failed to snapshot state\") {\n    time.Sleep(backoff)\n    resp, err = client.Nodes().UpdateStatus(req) // retry against another server\n}","preventionTips":["Monitor server memory and restart before OOM","Keep servers on supported, uniform versions","Alert on any state-store errors in server logs","Maintain regular state backups / raft snapshots"],"tags":["nomad","state-store","internal","snapshot"],"backgroundTag":"state-store-corruption","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}