{"record":{"id":"cf9472aa71a99d33","repo":"hashicorp/nomad","slug":"failed-to-find-system-jobs-for-s-v","errorCode":null,"errorMessage":"failed to find system jobs for '%s': %v","messagePattern":"failed to find system jobs for '(.+?)': (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"nomad/node_endpoint.go","lineNumber":1824,"sourceCode":"// Each Eval is scoped to a job, so we need to potentially trigger many evals.\nfunc (n *Node) createNodeEvals(node *structs.Node, nodeIndex uint64) ([]string, uint64, error) {\n\tnodeID := node.ID\n\n\t// Snapshot the state\n\tsnap, err := n.srv.fsm.State().Snapshot()\n\tif err != nil {\n\t\treturn nil, 0, fmt.Errorf(\"failed to snapshot state: %v\", err)\n\t}\n\n\t// Find all the allocations for this node\n\tallocs, err := snap.AllocsByNode(nil, nodeID)\n\tif err != nil {\n\t\treturn nil, 0, fmt.Errorf(\"failed to find allocs for '%s': %v\", nodeID, err)\n\t}\n\n\tsysJobsIter, err := snap.JobsByScheduler(nil, \"system\")\n\tif err != nil {\n\t\treturn nil, 0, fmt.Errorf(\"failed to find system jobs for '%s': %v\", nodeID, err)\n\t}\n\n\tvar sysJobs []*structs.Job\n\tfor jobI := sysJobsIter.Next(); jobI != nil; jobI = sysJobsIter.Next() {\n\t\tjob := jobI.(*structs.Job)\n\t\t// Avoid creating evals for jobs that don't run in this datacenter or\n\t\t// node pool. We could perform an entire feasibility check here, but\n\t\t// datacenter/pool is a good optimization to start with as their\n\t\t// cardinality tends to be low so the check shouldn't add much work.\n\t\t// If the job is stopped, skip it as well, otherwise we will create an\n\t\t// eval with state and broker overhead that will be an immediate no-op.\n\t\tif node.IsInPool(job.NodePool) && node.IsInAnyDC(job.Datacenters) && !job.Stopped() {\n\t\t\tsysJobs = append(sysJobs, job)\n\t\t}\n\t}\n\n\t// Fast-path if nothing to do\n\tif len(allocs) == 0 && len(sysJobs) == 0 {","sourceCodeStart":1806,"sourceCodeEnd":1842,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/node_endpoint.go#L1806-L1842","documentation":"createNodeEvals also reads all system jobs via snap.JobsByScheduler to decide which system-job evals must be re-created for the node. A failure in that state-store query is wrapped with the node ID. It is an internal server-side failure, not caused by the RPC payload.","triggerScenarios":"Any eval-creating RPC (Register, UpdateStatus, UpdateDrain, UpdateEligibility, Evaluate, deregister) when the JobsByScheduler(scheduler='system') query on the state snapshot errors.","commonSituations":"State store corruption or memdb index failures on an unhealthy server; rare, usually accompanied by other state-store errors in logs.","solutions":["Restart the affected Nomad server","Review server logs for the underlying state-store error message","Check for recently applied hot patches or version mismatch among servers","If persistent, restore the server's state from backup"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"resp, err := client.Nodes().UpdateStatus(req)\nif err != nil && strings.Contains(err.Error(), \"failed to find system jobs for\") {\n    // retry with backoff against another server\n}","preventionTips":["Maintain uniform server versions","Monitor server logs for state-store errors","Ensure adequate resources on server hosts","Restart servers showing persistent state errors"],"tags":["nomad","state-store","system-jobs","internal"],"backgroundTag":"state-store-corruption","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}