{"record":{"id":"0dbd3f316ccdb512","repo":"hashicorp/nomad","slug":"eval-delete-failed-v","errorCode":null,"errorMessage":"eval delete failed: %v","messagePattern":"eval delete failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":3643,"sourceCode":"\t\traw := iter.Next()\n\t\tif raw == nil {\n\t\t\tbreak\n\t\t}\n\t\teval := raw.(*structs.Evaluation)\n\t\tif eval.ID < pageToken {\n\t\t\tcontinue\n\t\t}\n\n\t\tdeleteOk, err := s.EvalIsUserDeleteSafe(nil, eval)\n\t\tif !deleteOk || err != nil {\n\t\t\tcontinue\n\t\t}\n\t\tmatch, err := filter.Evaluate(eval)\n\t\tif !match || err != nil {\n\t\t\tcontinue\n\t\t}\n\t\tif err := txn.Delete(\"evals\", eval); err != nil {\n\t\t\treturn fmt.Errorf(\"eval delete failed: %v\", err)\n\t\t}\n\t\tpageCount++\n\t}\n\n\terr = txn.Commit()\n\treturn err\n}\n\n// EvalIsUserDeleteSafe ensures an evaluation is safe to delete based on its\n// related allocation and job information. This follows similar, but different\n// rules to the eval reap checking, to ensure evaluations for running allocs or\n// allocs which need the evaluation detail are not deleted.\n//\n// Returns both a bool and an error so that error in querying the related\n// objects can be differentiated from reporting that the eval isn't safe to\n// delete.\nfunc (s *StateStore) EvalIsUserDeleteSafe(ws memdb.WatchSet, eval *structs.Evaluation) (bool, error) {\n","sourceCodeStart":3625,"sourceCodeEnd":3661,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L3625-L3661","documentation":"Nomad's StateStore wraps every low-level memdb transaction failure with a contextual message, and this one occurs in DeleteEvalsByFilter when txn.Delete(\"evals\", eval) fails inside the filtered batch delete. The wrapped error comes from the underlying memdb write transaction, which only fails on real store problems (table/index corruption, an object not matching the table schema, or a txn in an invalid state) — not because the eval is missing. It aborts the whole transaction before Commit, so no evals in the batch are removed.","triggerScenarios":"Calling the eval-delete-by-filter RPC/endpoint when the memdb evals table or its indexes are corrupted (e.g. snapshot restore of a bad snapshot), or an internal state store schema mismatch between the eval object being deleted and the table schema.","commonSituations":"Restored Raft snapshots on servers with version skew; disk-level corruption of the BoltDB state store; custom eval deletion tooling hitting servers mid-upgrade; operator-driven eval GC failing on a degraded server.","solutions":["Inspect the wrapped %v cause — it identifies the actual memdb/table problem; fix that root cause first","Verify state store integrity: compare the failing server against peers (nomad server members / Raft state) and replace the degraded server","Restore a known-good Raft snapshot if corruption is confirmed","Restart the server and let Raft replay/resync; if persistent, rejoin the server as a fresh member"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Before issuing the eval delete-by-filter request, verify server health and store integrity:\nhealth, _, _ := client.Agent().Health()\nif health.Server.Ok == false { return fmt.Errorf(\"server not healthy; skip eval delete\") }\n// Optionally dry-run the filter:\n// verify matching evals exist via client.Evals().List(nil) before deleting","typeGuard":"func isMemdbCorruption(err error) bool {\n\treturn err != nil && strings.Contains(err.Error(), \"object is not valid\")\n}","tryCatchPattern":"// Nomad Go client surfaces RPC errors as error; retry transient failures\nvar lastErr error\nfor i := 0; i < 3; i++ {\n\t_, _, err := client.Evals().DeleteByFilter(nil, filter) // or equivalent RPC\n\tif err == nil { break }\n\tif isMemdbCorruption(err) { return err } // do not retry corruption\n\tlastErr = err; time.Sleep(backoff(i))\n}","preventionTips":["Keep all server agents on the same Nomad version to avoid schema mismatches","Validate Raft snapshots before restoring them","Monitor server disk health and state store size","Run one-off eval GC during low load and verify server health first"],"tags":["nomad","state-store","memdb","evals"],"backgroundTag":"state-store-corruption","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}