{"record":{"id":"9ddd724ee99639d8","repo":"hashicorp/nomad","slug":"failed-to-get-blocked-evals-for-job-q-in-namespac","errorCode":null,"errorMessage":"failed to get blocked evals for job %q in namespace %q: %v","messagePattern":"failed to get blocked evals for job %q in namespace %q: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":3529,"sourceCode":"\n\t\t// Insert the job summary\n\t\tif hasSummaryChanged {\n\t\t\tjs.ModifyIndex = index\n\t\t\tif err := txn.Insert(\"job_summary\", js); err != nil {\n\t\t\t\treturn fmt.Errorf(\"job summary insert failed: %v\", err)\n\t\t\t}\n\t\t\tif err := txn.Insert(\"index\", &IndexEntry{\"job_summary\", index}); err != nil {\n\t\t\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t\t\t}\n\t\t}\n\t}\n\n\t// Check if the job has any blocked evaluations and cancel them\n\tif eval.Status == structs.EvalStatusComplete && len(eval.FailedTGAllocs) == 0 {\n\t\t// Get the blocked evaluation for a job if it exists\n\t\titer, err := txn.Get(\"evals\", \"job\", eval.Namespace, eval.JobID, structs.EvalStatusBlocked)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"failed to get blocked evals for job %q in namespace %q: %v\", eval.JobID, eval.Namespace, err)\n\t\t}\n\n\t\tvar blocked []*structs.Evaluation\n\t\tfor {\n\t\t\traw := iter.Next()\n\t\t\tif raw == nil {\n\t\t\t\tbreak\n\t\t\t}\n\t\t\tblocked = append(blocked, raw.(*structs.Evaluation))\n\t\t}\n\n\t\t// Go through and update the evals\n\t\tfor _, blockedEval := range blocked {\n\t\t\tnewEval := blockedEval.Copy()\n\t\t\tnewEval.Status = structs.EvalStatusCancelled\n\t\t\tnewEval.StatusDescription = fmt.Sprintf(\"evaluation %q successful\", eval.ID)\n\t\t\tnewEval.ModifyIndex = index\n\t\t\tnewEval.ModifyTime = eval.ModifyTime","sourceCodeStart":3511,"sourceCodeEnd":3547,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L3511-L3547","documentation":"Returned by UpsertEvals in Nomad's state store when the memdb query to fetch blocked evaluations for a completed job (via txn.Get on the \"evals\" table with the \"job\" index) fails. It wraps the underlying memdb lookup error and aborts the eval upsert transaction. It signals an index/table lookup problem, not merely 'no evals found'.","triggerScenarios":"txn.Get(\"evals\", \"job\", eval.Namespace, eval.JobID, structs.EvalStatusBlocked) returns an error while cancelling blocked evals for a completed eval with no FailedTGAllocs — typically an aborted txn or memdb internal failure.","commonSituations":"Server state store under memory pressure; corrupted in-memory DB after failed restore; rare memdb bugs during high eval throughput.","solutions":["Retry the operation (the wrapping Raft apply will run in a new transaction)","Check server logs for a prior memdb panic; restart the Nomad server to rebuild state from Raft/snapshot","Upgrade Nomad if a matching memdb bug is fixed in a newer release","Capture the wrapped inner error (%v) for a HashiCorp bug report if reproducible"],"exampleFix":"// caller side: treat as retryable state-store failure\n// before\nerr := stateStore.UpsertEvals(idx, evals)\n// after\nif err != nil && strings.HasPrefix(err.Error(), \"failed to get blocked evals\") {\n    // schedule retry of the raft apply\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Go\nif err := state.UpsertEvals(idx, evals); err != nil {\n    if strings.Contains(err.Error(), \"failed to get blocked evals\") {\n        return retry(fn) // fresh txn on retry\n    }\n    return err\n}","preventionTips":["Retry full Raft applies on state-store errors","Watch server logs for memdb panics","Restart servers promptly after failed snapshot restores","Avoid running servers at sustained memory limits"],"tags":["nomad","state-store","memdb","blocked-evals"],"backgroundTag":"state-store-lookup-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}