{"record":{"id":"414d96678c2d58b2","repo":"hashicorp/nomad","slug":"failed-to-get-job-s-v","errorCode":null,"errorMessage":"failed to get job '%s': %v","messagePattern":"failed to get job '(.+?)': (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"scheduler/scheduler_sysbatch.go","lineNumber":113,"sourceCode":"\t\t}\n\t\treturn err\n\t}\n\n\t// Update the status to complete\n\treturn setStatus(s.logger, s.planner, s.eval, nil,\n\t\ts.failedTGAllocs, s.planAnnotations, structs.EvalStatusComplete, \"\",\n\t\ts.queuedAllocs, \"\")\n}\n\n// process is wrapped in retryMax to iteratively run the handler until we have no\n// further work or we've made the maximum number of attempts.\nfunc (s *SysBatchScheduler) process() (bool, error) {\n\t// Lookup the Job by ID\n\tvar err error\n\tws := memdb.NewWatchSet()\n\ts.job, err = s.state.JobByID(ws, s.eval.Namespace, s.eval.JobID)\n\tif err != nil {\n\t\treturn false, fmt.Errorf(\"failed to get job '%s': %v\", s.eval.JobID, err)\n\t}\n\n\tnumTaskGroups := 0\n\tif !s.job.Stopped() {\n\t\tnumTaskGroups = len(s.job.TaskGroups)\n\t}\n\ts.queuedAllocs = make(map[string]int, numTaskGroups)\n\n\t// Get the ready nodes in the required datacenters\n\tif !s.job.Stopped() {\n\t\ts.nodes, s.notReadyNodes, s.nodesByDC, err = readyNodesInDCsAndPool(\n\t\t\ts.state, s.job.Datacenters, s.job.NodePool)\n\t\tif err != nil {\n\t\t\treturn false, fmt.Errorf(\"failed to get ready nodes: %v\", err)\n\t\t}\n\t}\n\n\t// Create a plan","sourceCodeStart":95,"sourceCodeEnd":131,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/scheduler/scheduler_sysbatch.go#L95-L131","documentation":"During SysBatchScheduler.process, the state store lookup JobByID for the evaluation's namespace/JobID failed, so the scheduler cannot load the job it is supposed to evaluate. The wrapped %v contains the underlying state-store error (not 'job not found' — a missing job returns nil, nil and is handled separately).","triggerScenarios":"process() calls s.state.JobByID(ws, s.eval.Namespace, s.eval.JobID) and receives err != nil — state store backend failure, memdb/watch-set error, or I/O problem on the server.","commonSituations":"Degraded Nomad server state store; raft/IO errors during evaluation; memory pressure or corruption on the leader.","solutions":["Read the wrapped cause in the error and check the Nomad server logs for the underlying state-store failure.","Verify the job still exists in the expected namespace (nomad job status -namespace <ns> <job>).","Confirm the server's raft/state store is healthy; restart the affected server if needed.","Re-submit or trigger a new evaluation (nomad job eval) once the state store recovers."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"ws := memdb.NewWatchSet()\nif _, err := state.JobByID(ws, ns, jobID); err != nil {\n    return fmt.Errorf(\"job %s/%s not readable: %w\", ns, jobID, err)\n}","typeGuard":"func jobReadable(s structs.State, ns, jobID string) bool {\n    _, err := s.JobByID(memdb.NewWatchSet(), ns, jobID)\n    return err == nil\n}","tryCatchPattern":"if err := sched.Process(eval); err != nil {\n    if strings.Contains(err.Error(), \"failed to get job '\") {\n        return retryWithBackoff(func() error { return sched.Process(eval) })\n    }\n    return err\n}","preventionTips":["Confirm the job exists in the intended namespace before evaluating.","Monitor server state-store health and disk usage.","Retry evaluations on transient state-store errors.","Keep Nomad servers on a supported, patched version."],"tags":["nomad","scheduler","sysbatch","state-store"],"backgroundTag":"state-store-read-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}