{"record":{"id":"e21847cd5384fa8a","repo":"hashicorp/nomad","slug":"upserting-evals-failed-v","errorCode":null,"errorMessage":"upserting evals failed: %v","messagePattern":"upserting evals failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":4063,"sourceCode":"\tpopulatedAllocs := []*structs.Allocation{}\n\t// Handle each of the updated allocations\n\tfor _, a := range allocs {\n\t\tnodeIDs.Insert(a.NodeID)\n\t\tca, err := s.nestedUpdateAllocFromClient(txn, index, a)\n\t\tif ca == nil {\n\t\t\tcontinue\n\t\t}\n\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"updating alloc failed: %v\", err)\n\t\t}\n\t\tpopulatedAllocs = append(populatedAllocs, ca)\n\t}\n\n\tif len(req.Evals) > 0 {\n\t\terr := s.UpsertEvalsTxn(index, evals, txn)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"upserting evals failed: %v\", err)\n\t\t}\n\t}\n\n\tjobs := map[structs.NamespacedID]string{}\n\n\tfor _, alloc := range populatedAllocs {\n\t\ttuple := structs.NamespacedID{\n\t\t\tID:        alloc.JobID,\n\t\t\tNamespace: alloc.Namespace,\n\t\t}\n\n\t\tjobs[tuple] = \"\"\n\t}\n\n\tif err := s.setJobStatuses(index, txn, jobs, false); err != nil {\n\t\treturn fmt.Errorf(\"setting job status failed: %v\", err)\n\t}\n","sourceCodeStart":4045,"sourceCodeEnd":4081,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L4045-L4081","documentation":"Thrown during client alloc update processing when s.UpsertEvalsTxn fails to write the evaluations accompanying the alloc update. The evals are upserted in the same transaction as the allocs, so any eval write error rolls back the entire update batch.","triggerScenarios":"Node.UpdateAllocs or UpdateEval requests carrying req.Evals where UpsertEvalsTxn hits a memdb write/index error for the evals table.","commonSituations":"Server-side failures while persisting follow-up evals created by alloc updates; seen in server logs during state store contention or corruption.","solutions":["Inspect the wrapped UpsertEvalsTxn error cause (often the per-eval index update).","Check server logs and Raft apply health.","Restart the Nomad server if the store is wedged.","Clients re-drive evals, so no manual resubmission is usually needed once the server is healthy."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"if strings.Contains(err.Error(), \"upserting evals failed\") {\n  // whole txn rolled back; safe to retry once server recovers\n  retryWithBackoff(func() error { return upsertAllocsAndEvals(req) })\n}","preventionTips":["Rely on transactional rollback: the alloc batch is atomic, retry the whole batch.","Keep eval volumes within Nomad's tested limits per update.","Monitor Raft apply latency.","Upgrade Nomad if the underlying UpsertEvalsTxn cause matches a known bug."],"tags":["nomad","state-store","evals"],"backgroundTag":"state-store-write-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}