{"record":{"id":"a554c7abf67af085","repo":"hashicorp/nomad","slug":"failed-to-lookup-job-for-eval-v","errorCode":null,"errorMessage":"failed to lookup job for eval: %v","messagePattern":"failed to lookup job for eval: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":3664,"sourceCode":"\t}\n\n\terr = txn.Commit()\n\treturn err\n}\n\n// EvalIsUserDeleteSafe ensures an evaluation is safe to delete based on its\n// related allocation and job information. This follows similar, but different\n// rules to the eval reap checking, to ensure evaluations for running allocs or\n// allocs which need the evaluation detail are not deleted.\n//\n// Returns both a bool and an error so that error in querying the related\n// objects can be differentiated from reporting that the eval isn't safe to\n// delete.\nfunc (s *StateStore) EvalIsUserDeleteSafe(ws memdb.WatchSet, eval *structs.Evaluation) (bool, error) {\n\n\tjob, err := s.JobByID(ws, eval.Namespace, eval.JobID)\n\tif err != nil {\n\t\treturn false, fmt.Errorf(\"failed to lookup job for eval: %v\", err)\n\t}\n\n\tallocs, err := s.AllocsByEval(ws, eval.ID)\n\tif err != nil {\n\t\treturn false, fmt.Errorf(\"failed to lookup eval allocs: %v\", err)\n\t}\n\n\treturn isEvalDeleteSafe(allocs, job), nil\n}\n\nfunc isEvalDeleteSafe(allocs []*structs.Allocation, job *structs.Job) bool {\n\n\t// If the job is deleted, stopped, or dead, all allocs are terminal and\n\t// the eval can be deleted.\n\tif job == nil || job.Stop || job.Status == structs.JobStatusDead {\n\t\treturn true\n\t}\n","sourceCodeStart":3646,"sourceCodeEnd":3682,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L3646-L3682","documentation":"StateStore.EvalIsUserDeleteSafe wraps any failure from s.JobByID when resolving the job referenced by the eval being checked for user-initiated deletion safety. It means the job lookup against the jobs table returned a transaction-level error (not merely 'not found' — a nil job is handled by isEvalDeleteSafe logic downstream). The caller gets a wrapped error and a false result, so the delete is treated as unsafe.","triggerScenarios":"Calling EvalIsUserDeleteSafe (via the eval delete user-safety check RPC) when the memdb jobs table transaction errors — schema/index corruption or a broken write transaction at read time.","commonSituations":"State store corruption after a failed snapshot restore; evaluating deletion of an eval whose job lives in a namespace being concurrently purged; version-skewed clusters where the jobs table schema changed.","solutions":["Inspect the wrapped cause for the underlying memdb error and fix the store-level root cause","Confirm the eval's Namespace/JobID fields are valid; a corrupted eval record can make the lookup fail","Check for recent snapshot restores or upgrades; restore a clean snapshot or roll back the version skew","Retry the deletion check after the server's state store has been verified/replaced"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Verify the job exists and the eval is well-formed before the delete-safety check\neval, _, err := client.Evals().GetEval(evalID, nil)\nif err != nil { return err }\njob, _, err := client.Jobs().Info(eval.Namespace, eval.JobID, nil)\nif err != nil { return fmt.Errorf(\"job %s/%s not found: %w\", eval.Namespace, eval.JobID, err) }","typeGuard":"func evalJobLookupFailed(err error) bool {\n\treturn err != nil && strings.Contains(err.Error(), \"failed to lookup job for eval\")\n}","tryCatchPattern":"if err != nil {\n\tif evalJobLookupFailed(err) {\n\t\t// store-level problem: verify server health before retrying\n\t\treturn fmt.Errorf(\"state store job lookup failed; check server health: %w\", err)\n\t}\n\treturn err\n}","preventionTips":["Check the eval's Namespace and JobID are populated before deletion flows","Align Nomad versions across servers before upgrades","Validate snapshots before restore","Watch server state-store logs for memdb errors"],"tags":["nomad","state-store","evals","job-lookup"],"backgroundTag":"state-store-corruption","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}