{"record":{"id":"90fa92f5987c7a8d","repo":"hashicorp/nomad","slug":"failed-to-get-tainted-nodes-for-job-s-v-90fa92","errorCode":null,"errorMessage":"failed to get tainted nodes for job '%s': %v","messagePattern":"failed to get tainted nodes for job '(.+?)': (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"scheduler/scheduler_sysbatch.go","lineNumber":223,"sourceCode":"\ts.stack.SetJob(job)\n\ts.stack.SetSchedulerConfiguration(schedConfig.WithNodePool(pool))\n\treturn nil\n}\n\n// computeJobAllocs is used to reconcile differences between the job,\n// existing allocations and node status to update the allocations.\nfunc (s *SysBatchScheduler) computeJobAllocs() error {\n\t// Lookup the allocations by JobID\n\tws := memdb.NewWatchSet()\n\tallocs, err := s.state.AllocsByJob(ws, s.eval.Namespace, s.eval.JobID, true)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to get allocs for job '%s': %v\", s.eval.JobID, err)\n\t}\n\n\t// Determine the tainted nodes containing job allocs\n\ttainted, err := taintedNodes(s.state, allocs)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to get tainted nodes for job '%s': %v\", s.eval.JobID, err)\n\t}\n\n\t// Update the allocations which are in pending/running state on tainted\n\t// nodes to lost.\n\tupdateNonTerminalAllocsToLost(s.plan, tainted, allocs)\n\n\t// Split out terminal allocations\n\tlive, term := structs.SplitTerminalAllocs(allocs)\n\n\t// Diff the required and existing allocations\n\tnr := reconciler.NewNodeReconciler(nil)\n\tr := nr.Compute(s.job, s.nodes, s.notReadyNodes, tainted, live, term)\n\tif s.logger.IsDebug() {\n\t\ts.logger.Debug(\"reconciled current state with desired state\", r.Fields()...)\n\t}\n\n\t// Add all the allocs to stop\n\tfor _, e := range r.Stop {","sourceCodeStart":205,"sourceCodeEnd":241,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/scheduler/scheduler_sysbatch.go#L205-L241","documentation":"After fetching the job's allocations, SysBatchScheduler.computeJobAllocs calls taintedNodes to identify dead/drained nodes holding those allocs so non-terminal allocs can be marked lost; this error wraps a failure of the taintedNodes state-store lookups. Without the tainted-node set the scheduler cannot correctly update allocs on failed nodes.","triggerScenarios":"process() -> computeJobAllocs(): taintedNodes(s.state, allocs) returns err != nil — NodeByID state-store lookup failure for one of the job's allocation node IDs.","commonSituations":"State-store errors while resolving nodes referenced by allocations; nodes deregistered/purged while state store is degraded; raft/disk failures on the leader.","solutions":["Inspect the wrapped cause and server logs to find the failing node lookup / state-store error.","Check that node IDs referenced by the job's allocs exist and are consistent (nomad node status).","Repair or restart the Nomad server state store; restore from backup if corrupt.","Re-run the evaluation after recovery (nomad job eval)."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// preflight: every node referenced by the job's allocs is resolvable\nfor _, a := range allocs {\n    if _, err := state.NodeByID(nil, a.NodeID); err != nil {\n        return fmt.Errorf(\"node %s unreadable: %w\", a.NodeID, err)\n    }\n}","typeGuard":"func taintedNodesResolvable(s structs.State, allocs []*structs.Allocation) bool {\n    _, err := taintedNodes(s, allocs)\n    return err == nil\n}","tryCatchPattern":"if err := sched.Process(eval); err != nil {\n    if strings.Contains(err.Error(), \"failed to get tainted nodes\") {\n        return retryWithBackoff(func() error { return sched.Process(eval) })\n    }\n    return err\n}","preventionTips":["Avoid purging node records while allocs reference them.","Monitor state-store health on Nomad servers.","Retry evaluations after transient state-store failures.","Keep the cluster on a supported Nomad version."],"tags":["nomad","scheduler","sysbatch","tainted-nodes","state-store"],"backgroundTag":"state-store-read-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}