{"record":{"id":"d17e540194b07ac4","repo":"hashicorp/nomad","slug":"failed-to-get-allocs-for-job-s-v-d17e54","errorCode":null,"errorMessage":"failed to get allocs for job '%s': %v","messagePattern":"failed to get allocs for job '(.+?)': (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"scheduler/scheduler_sysbatch.go","lineNumber":217,"sourceCode":"\n\t_, schedConfig, err := s.state.SchedulerConfig()\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to get scheduler configuration: %v\", err)\n\t}\n\n\ts.stack.SetJob(job)\n\ts.stack.SetSchedulerConfiguration(schedConfig.WithNodePool(pool))\n\treturn nil\n}\n\n// computeJobAllocs is used to reconcile differences between the job,\n// existing allocations and node status to update the allocations.\nfunc (s *SysBatchScheduler) computeJobAllocs() error {\n\t// Lookup the allocations by JobID\n\tws := memdb.NewWatchSet()\n\tallocs, err := s.state.AllocsByJob(ws, s.eval.Namespace, s.eval.JobID, true)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to get allocs for job '%s': %v\", s.eval.JobID, err)\n\t}\n\n\t// Determine the tainted nodes containing job allocs\n\ttainted, err := taintedNodes(s.state, allocs)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to get tainted nodes for job '%s': %v\", s.eval.JobID, err)\n\t}\n\n\t// Update the allocations which are in pending/running state on tainted\n\t// nodes to lost.\n\tupdateNonTerminalAllocsToLost(s.plan, tainted, allocs)\n\n\t// Split out terminal allocations\n\tlive, term := structs.SplitTerminalAllocs(allocs)\n\n\t// Diff the required and existing allocations\n\tnr := reconciler.NewNodeReconciler(nil)\n\tr := nr.Compute(s.job, s.nodes, s.notReadyNodes, tainted, live, term)","sourceCodeStart":199,"sourceCodeEnd":235,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/scheduler/scheduler_sysbatch.go#L199-L235","documentation":"SysBatchScheduler.computeJobAllocs queries AllocsByJob (all allocations for the eval's namespace/JobID, including terminal ones) and wraps a failure of that state-store read in this error. The scheduler needs the existing allocs to compute placements and stop/lost updates for sysbatch jobs.","triggerScenarios":"process() -> computeJobAllocs(): s.state.AllocsByJob(ws, s.eval.Namespace, s.eval.JobID, true) returns err != nil due to a state-store backend failure.","commonSituations":"State-store read errors on a Nomad server under raft/disk failure; memory pressure in memdb; errors during leader failover while a sysbatch eval is being processed.","solutions":["Inspect the wrapped cause and Nomad server logs for the underlying state-store error.","Verify raft/state store health; restart or replace the failing server.","Check alloc table integrity if corruption is suspected (nomad operator debug).","Re-trigger the evaluation once the state store is healthy."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"ws := memdb.NewWatchSet()\nif _, err := state.AllocsByJob(ws, ns, jobID, true); err != nil {\n    return fmt.Errorf(\"allocs for %s/%s unreadable: %w\", ns, jobID, err)\n}","typeGuard":"func allocsReadable(s structs.State, ns, jobID string) bool {\n    _, err := s.AllocsByJob(memdb.NewWatchSet(), ns, jobID, true)\n    return err == nil\n}","tryCatchPattern":"if err := sched.Process(eval); err != nil {\n    if strings.Contains(err.Error(), \"failed to get allocs for job\") {\n        return retryWithBackoff(func() error { return sched.Process(eval) })\n    }\n    return err\n}","preventionTips":["Monitor Nomad server memory and state-store health (large alloc tables).","Run nomad operator debug if corruption is suspected.","Retry evaluations during leader failover windows.","Keep servers patched and the data directory on reliable storage."],"tags":["nomad","scheduler","sysbatch","allocations","state-store"],"backgroundTag":"state-store-read-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}