{"record":{"id":"8345bfbac1080770","repo":"hashicorp/nomad","slug":"failed-to-lookup-evals-v","errorCode":null,"errorMessage":"failed to lookup evals: %v","messagePattern":"failed to lookup evals: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"nomad/state/state_store.go","lineNumber":3614,"sourceCode":"\tdefer txn.Abort()\n\n\t// These are always user-initiated, so ensure the eval broker is paused.\n\t_, schedConfig, err := s.schedulerConfigTxn(txn)\n\tif err != nil {\n\t\treturn err\n\t}\n\tif schedConfig == nil || !schedConfig.PauseEvalBroker {\n\t\treturn errors.New(\"eval broker is enabled; eval broker must be paused to delete evals\")\n\t}\n\n\tfilter, err := bexpr.CreateEvaluator(filterExpr)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\titer, err := s.Evals(nil, SortDefault)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to lookup evals: %v\", err)\n\t}\n\n\t// Note: Paginator imports this package for testing so we can't just use\n\t// Paginator\n\tpageCount := int32(0)\n\n\tfor {\n\t\tif pageCount >= perPage {\n\t\t\tbreak\n\t\t}\n\t\traw := iter.Next()\n\t\tif raw == nil {\n\t\t\tbreak\n\t\t}\n\t\teval := raw.(*structs.Evaluation)\n\t\tif eval.ID < pageToken {\n\t\t\tcontinue\n\t\t}","sourceCodeStart":3596,"sourceCodeEnd":3632,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L3596-L3632","documentation":"Returned by DeleteEvalsByFilter (eval garbage collection) when iterating all evaluations via s.Evals(nil, SortDefault) fails. The wrapped memdb iterator error aborts the GC pass. It is a full-table scan failure inside the state store, not a filter-evaluation problem.","triggerScenarios":"s.Evals(nil, SortDefault) errors during DeleteEvalsByFilter — typically an aborted transaction, memdb internal error, or resource exhaustion while building the eval iterator during GC.","commonSituations":"Eval GC on clusters with very large eval backlogs; memory pressure during compaction; server recovering after failover or snapshot restore.","solutions":["Wait for the next GC cycle; the periodic GC will retry automatically","Reduce eval backlog manually (nomad system gc) and monitor server memory","Restart servers to rebuild state if the error is persistent","Upgrade Nomad; report with the wrapped inner error if reproducible"],"exampleFix":"// before\nerr := s.DeleteEvalsByFilter(...)\n// after\nif err != nil && strings.Contains(err.Error(), \"failed to lookup evals\") {\n    // skip this GC round; retry on next tick\n    return nil\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Go\nif err := s.DeleteEvalsByFilter(...); err != nil {\n    if strings.Contains(err.Error(), \"failed to lookup evals\") {\n        time.Sleep(gcInterval); return s.DeleteEvalsByFilter(...)\n    }\n    return err\n}","preventionTips":["Run `nomad system gc` during low-traffic windows to shrink eval backlogs","Ensure GC thresholds keep eval table sizes manageable","Watch server memory during compaction/GC","Keep servers on patched Nomad releases"],"tags":["nomad","state-store","memdb","garbage-collection"],"backgroundTag":"state-store-lookup-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}