{"record":{"id":"99837821d3d563b8","repo":"hashicorp/nomad","slug":"failed-to-determine-last-evaluation-index-for-job","errorCode":null,"errorMessage":"failed to determine last evaluation index for job %q: %v","messagePattern":"failed to determine last evaluation index for job %q: (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/deploymentwatcher/deployment_watcher.go","lineNumber":608,"sourceCode":"\tcreateEval        bool\n\tfailDeployment    bool\n\trollback          bool\n\tallowReplacements []string\n}\n\n// handleAllocUpdate is used to compute the set of actions to take based on the\n// updated allocations for the deployment.\nfunc (w *deploymentWatcher) handleAllocUpdate(allocs []*structs.AllocListStub) (allocUpdateResult, error) {\n\tvar res allocUpdateResult\n\n\t// Get the latest evaluation index\n\tlatestEval, err := w.jobEvalStatus()\n\tif err != nil {\n\t\tif err == context.Canceled || w.ctx.Err() == context.Canceled {\n\t\t\treturn res, err\n\t\t}\n\n\t\treturn res, fmt.Errorf(\"failed to determine last evaluation index for job %q: %v\", w.j.ID, err)\n\t}\n\n\tdeployment := w.getDeployment()\n\tfor _, alloc := range allocs {\n\t\tdstate, ok := deployment.TaskGroups[alloc.TaskGroup]\n\t\tif !ok {\n\t\t\tcontinue\n\t\t}\n\n\t\t// Check if we can already fail the deployment\n\t\tfailDeployment := w.shouldFailEarly(deployment, alloc, dstate)\n\n\t\t// Check if the allocation has failed and we need to mark it for allow\n\t\t// replacements\n\t\tif alloc.DeploymentStatus.IsUnhealthy() && !failDeployment &&\n\t\t\tdeployment.Active() && !alloc.DesiredTransition.ShouldReschedule() {\n\t\t\tres.allowReplacements = append(res.allowReplacements, alloc.ID)\n\t\t\tcontinue","sourceCodeStart":590,"sourceCodeEnd":626,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/deploymentwatcher/deployment_watcher.go#L590-L626","documentation":"Inside the deployment watcher, when an allocation update arrives, the watcher needs the job's latest evaluation to know the current deployment state. If jobEvalStatus (a state-store read) fails, the alloc update is dropped and this wrapped error is returned; the watcher loop then retries via the watch loop.","triggerScenarios":"State store read errors for the job's evaluations while processing alloc updates — typically during Raft instability, FSM/state-store restore, or a context cancellation racing the lookup (cancellation is specially handled and passes through).","commonSituations":"Leader failover mid-deployment; server under heavy load with state store contention; cluster restore where the job was GC'd between deployment creation and alloc updates.","solutions":["Check server logs around the error for the underlying state-store/Raft cause","If it followed a leader election, it is usually transient — the watcher re-runs; verify the deployment progresses with `nomad deployment status`","If persistent, inspect `nomad server members` for Raft health and restart/replace the failing server"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// caller-side: monitor server health before/while deployments run\nnomad server members   # ensure the leader is stable and Raft is healthy\nnomad deployment status <id>  # confirm progress despite transient watcher errors","typeGuard":null,"tryCatchPattern":"// server-side watcher already retries via its watch loop; client-side, poll until settled:\nfor i := 0; i < 10; i++ {\n    d, _, err := client.Deployments().Info(deployID, nil)\n    if err == nil && (d.Status == \"successful\" || d.Status == \"failed\" || d.Status == \"cancelled\") { return d, nil }\n    time.Sleep(2 * time.Second)\n}\nreturn nil, errors.New(\"deployment did not settle\")","preventionTips":["Watch for this error in server logs during leader elections; it is normally self-healing","Keep the cluster's Raft healthy (odd number of servers, stable leadership)","Avoid rapid job GC settings that remove jobs mid-deployment","If persistent, restart the affected server and verify the deployment resumes"],"tags":["nomad","deployment-watcher","internal","state-store"],"backgroundTag":"state-store-read-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}