{"record":{"id":"673aec8e9e249b4f","repo":"hashicorp/nomad","slug":"failed-to-lookup-eval-allocs-v","errorCode":null,"errorMessage":"failed to lookup eval allocs: %v","messagePattern":"failed to lookup eval allocs: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":3669,"sourceCode":"\n// EvalIsUserDeleteSafe ensures an evaluation is safe to delete based on its\n// related allocation and job information. This follows similar, but different\n// rules to the eval reap checking, to ensure evaluations for running allocs or\n// allocs which need the evaluation detail are not deleted.\n//\n// Returns both a bool and an error so that error in querying the related\n// objects can be differentiated from reporting that the eval isn't safe to\n// delete.\nfunc (s *StateStore) EvalIsUserDeleteSafe(ws memdb.WatchSet, eval *structs.Evaluation) (bool, error) {\n\n\tjob, err := s.JobByID(ws, eval.Namespace, eval.JobID)\n\tif err != nil {\n\t\treturn false, fmt.Errorf(\"failed to lookup job for eval: %v\", err)\n\t}\n\n\tallocs, err := s.AllocsByEval(ws, eval.ID)\n\tif err != nil {\n\t\treturn false, fmt.Errorf(\"failed to lookup eval allocs: %v\", err)\n\t}\n\n\treturn isEvalDeleteSafe(allocs, job), nil\n}\n\nfunc isEvalDeleteSafe(allocs []*structs.Allocation, job *structs.Job) bool {\n\n\t// If the job is deleted, stopped, or dead, all allocs are terminal and\n\t// the eval can be deleted.\n\tif job == nil || job.Stop || job.Status == structs.JobStatusDead {\n\t\treturn true\n\t}\n\n\t// Iterate the allocations associated to the eval, if any, and check\n\t// whether we can delete the eval.\n\tfor _, alloc := range allocs {\n\n\t\t// If the allocation is still classed as running on the client, or","sourceCodeStart":3651,"sourceCodeEnd":3687,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L3651-L3687","documentation":"StateStore.EvalIsUserDeleteSafe wraps failures from s.AllocsByEval when fetching allocations belonging to the eval being checked. This is a read-transaction failure in the allocs table, not simply 'no allocs exist' (an empty result is valid and safe). The eval delete safety check aborts and reports the eval as not safe to delete.","triggerScenarios":"Calling the eval delete safety check when the allocs-by-eval memdb index is corrupted or the read transaction fails mid-lookup (schema mismatch, index rebuild failure).","commonSituations":"Corrupted allocs index after a bad snapshot restore; deleting evals for jobs whose allocations were written by an older Nomad schema version; disk corruption on the server's state store.","solutions":["Inspect the wrapped cause to identify the allocs table/index problem","Verify allocs index integrity; restore a known-good Raft snapshot if corruption is confirmed","Upgrade/align all servers to the same Nomad version to avoid schema mismatches","Retry after replacing or restarting the degraded server"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Pre-check allocs for the eval via the API before the safety check\nallocs, _, err := client.Allocs().List(nil)\nif err != nil { return err }\nfor _, a := range allocs {\n\tif a.EvalID == evalID { /* allocs exist; safety check will evaluate them */ }\n}","typeGuard":"func evalAllocLookupFailed(err error) bool {\n\treturn err != nil && strings.Contains(err.Error(), \"failed to lookup eval allocs\")\n}","tryCatchPattern":"if err != nil {\n\tif evalAllocLookupFailed(err) {\n\t\treturn fmt.Errorf(\"allocs table read failed; verify server state store: %w\", err)\n\t}\n\treturn err\n}","preventionTips":["Keep servers version-aligned so allocs schema/indexes match","Restore only validated snapshots","Monitor allocs table health and index rebuild errors","Avoid manual state store edits or external DB writes"],"tags":["nomad","state-store","allocs","evals"],"backgroundTag":"state-store-corruption","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}