{"record":{"id":"230c9b785abaf790","repo":"hashicorp/nomad","slug":"setting-job-status-failed-v","errorCode":null,"errorMessage":"setting job status failed: %v","messagePattern":"setting job status failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":3469,"sourceCode":"// in a transaction.  Useful for when making multiple modifications atomically.\nfunc (s *StateStore) UpsertEvalsTxn(index uint64, evals []*structs.Evaluation, txn Txn) error {\n\t// Do a nested upsert\n\tjobs := make(map[structs.NamespacedID]string, len(evals))\n\tfor _, eval := range evals {\n\t\tif err := s.nestedUpsertEval(txn, index, eval); err != nil {\n\t\t\treturn err\n\t\t}\n\n\t\ttuple := structs.NamespacedID{\n\t\t\tID:        eval.JobID,\n\t\t\tNamespace: eval.Namespace,\n\t\t}\n\t\tjobs[tuple] = \"\"\n\t}\n\n\t// Set the job's status\n\tif err := s.setJobStatuses(index, txn, jobs, false); err != nil {\n\t\treturn fmt.Errorf(\"setting job status failed: %v\", err)\n\t}\n\n\treturn nil\n}\n\n// nestedUpsertEvaluation is used to nest an evaluation upsert within a transaction\nfunc (s *StateStore) nestedUpsertEval(txn *txn, index uint64, eval *structs.Evaluation) error {\n\t// Lookup the evaluation\n\texisting, err := txn.First(\"evals\", \"id\", eval.ID)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"eval lookup failed: %v\", err)\n\t}\n\n\t// Update the indexes\n\tif existing != nil {\n\t\teval.CreateIndex = existing.(*structs.Evaluation).CreateIndex\n\t\teval.ModifyIndex = index\n\t} else {","sourceCodeStart":3451,"sourceCodeEnd":3487,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L3451-L3487","documentation":"Returned by StateStore.UpsertEvals (after eval insert) when the follow-up s.setJobStatuses call fails while recomputing and setting the statuses of jobs affected by the evaluations. The evals themselves may be staged, but because job status update failed the transaction is abandoned and this wrapper error is what callers see.","triggerScenarios":"Calling UpsertEvals where setJobStatuses fails internally — e.g. it cannot look up the job(s) referenced by the evals, or the internal job-status insert/update into memdb errors (job missing from state, or memdb write failure).","commonSituations":"Evals referencing jobs that were deregistered concurrently (race between eval creation and job delete); jobs in a different namespace than expected; state store write failures under memory pressure.","solutions":["Check the wrapped inner error from setJobStatuses in the message","Verify the evals reference existing jobs (namespace + job ID) in state","Retry the operation; if evals for deleted jobs keep arriving, investigate the scheduler/eval producer","Restart servers if the inner error indicates memdb corruption"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Ensure every eval references a live job before UpsertEvals\nfor _, e := range evals {\n\tjob, err := state.JobByID(ws, e.Namespace, e.JobID)\n\tif err != nil { return err }\n\tif job == nil { /* drop or re-create eval */ }\n}","typeGuard":"if ev, ok := raw.(*structs.Evaluation); ok { /* use ev */ }","tryCatchPattern":"if err := state.UpsertEvals(idx, evals); err != nil {\n\tif strings.Contains(err.Error(), \"setting job status failed\") {\n\t\t// inspect inner cause: missing job vs memdb write failure\n\t}\n\treturn err\n}","preventionTips":["Reconcile evals against current jobs before upsert","Avoid racing job deregistration with eval creation","Log the wrapped inner error for root-cause triage"],"tags":["nomad","state-store","evals","job-status"],"backgroundTag":"state-store-write-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}