{"record":{"id":"5d0c32a3eeb0f492","repo":"hashicorp/nomad","slug":"failed-to-get-ready-nodes-v","errorCode":null,"errorMessage":"failed to get ready nodes: %v","messagePattern":"failed to get ready nodes: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"scheduler/scheduler_sysbatch.go","lineNumber":127,"sourceCode":"\tvar err error\n\tws := memdb.NewWatchSet()\n\ts.job, err = s.state.JobByID(ws, s.eval.Namespace, s.eval.JobID)\n\tif err != nil {\n\t\treturn false, fmt.Errorf(\"failed to get job '%s': %v\", s.eval.JobID, err)\n\t}\n\n\tnumTaskGroups := 0\n\tif !s.job.Stopped() {\n\t\tnumTaskGroups = len(s.job.TaskGroups)\n\t}\n\ts.queuedAllocs = make(map[string]int, numTaskGroups)\n\n\t// Get the ready nodes in the required datacenters\n\tif !s.job.Stopped() {\n\t\ts.nodes, s.notReadyNodes, s.nodesByDC, err = readyNodesInDCsAndPool(\n\t\t\ts.state, s.job.Datacenters, s.job.NodePool)\n\t\tif err != nil {\n\t\t\treturn false, fmt.Errorf(\"failed to get ready nodes: %v\", err)\n\t\t}\n\t}\n\n\t// Create a plan\n\ts.plan = s.eval.MakePlan(s.job)\n\n\t// Reset the failed allocations\n\ts.failedTGAllocs = nil\n\n\t// Create an evaluation context\n\ts.ctx = feasible.NewEvalContext(s.eventsCh, s.state, s.plan, s.logger)\n\n\t// Construct the placement stack\n\ts.stack = feasible.NewSystemStack(true, s.ctx)\n\tif !s.job.Stopped() {\n\t\ts.setJob(s.job)\n\t}\n","sourceCodeStart":109,"sourceCodeEnd":145,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/scheduler/scheduler_sysbatch.go#L109-L145","documentation":"SysBatchScheduler.process calls readyNodesInDCsAndPool to list ready nodes in the job's datacenters and node pool; this error wraps any failure from that state-store query. It means the scheduler could not determine the eligible client nodes for placement.","triggerScenarios":"readyNodesInDCsAndPool(s.state, job.Datacenters, job.NodePool) returns err != nil — failure iterating nodes from the state store (backend error), not merely zero ready nodes (which is handled without error).","commonSituations":"State-store read failure on the Nomad server; node pool / datacenter metadata inconsistency after cluster restore; underlying raft errors during evaluation.","solutions":["Inspect the wrapped cause and server logs for the underlying state-store error.","Verify node pool and datacenter configuration on the job matches the cluster (nomad node status, nomad node pool list).","Check the health of the Nomad server state store and raft log; restart the server if corrupt.","Trigger a new evaluation once the state store is healthy."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// preflight: node listing works and job's DCs/pool are sane\nif _, _, _, err := readyNodesInDCsAndPool(state, job.Datacenters, job.NodePool); err != nil {\n    return fmt.Errorf(\"node query preflight failed: %w\", err)\n}","typeGuard":"func nodesQueryable(s structs.State, dcs []string, pool string) bool {\n    _, _, _, err := readyNodesInDCsAndPool(s, dcs, pool)\n    return err == nil\n}","tryCatchPattern":"if err := sched.Process(eval); err != nil {\n    if strings.Contains(err.Error(), \"failed to get ready nodes\") {\n        return retryWithBackoff(func() error { return sched.Process(eval) })\n    }\n    return err\n}","preventionTips":["Validate job datacenters/node pool against actual cluster nodes before submit.","Watch state-store and raft health on Nomad servers.","Retry evaluations on transient state-store failures.","Keep node pool definitions consistent across the cluster."],"tags":["nomad","scheduler","sysbatch","nodes","state-store"],"backgroundTag":"state-store-read-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}