{"record":{"id":"67f48ae94ab4e65b","repo":"hashicorp/nomad","slug":"eval-broker-is-enabled-eval-broker-must-be-paused","errorCode":null,"errorMessage":"eval broker is enabled; eval broker must be paused to delete evals","messagePattern":"eval broker is enabled; eval broker must be paused to delete evals","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/eval_endpoint.go","lineNumber":484,"sourceCode":"\t}\n\n\tif args.Filter != \"\" && !e.srv.peersCache.ServersMeetMinimumVersion(\n\t\te.srv.Region(), minVersionEvalDeleteByFilter, true) {\n\t\treturn fmt.Errorf(\n\t\t\t\"all servers must be running version %v or later to delete evals by filter\",\n\t\t\tminVersionEvalDeleteByFilter)\n\t}\n\tif args.Filter != \"\" && len(args.EvalIDs) > 0 {\n\t\treturn fmt.Errorf(\"evals cannot be deleted by both ID and filter\")\n\t}\n\tif args.Filter == \"\" && len(args.EvalIDs) == 0 {\n\t\treturn fmt.Errorf(\"evals must be deleted by either ID or filter\")\n\t}\n\n\t// The eval broker must be disabled otherwise Nomad's state will likely get\n\t// wild in a very un-fun way.\n\tif e.srv.evalBroker.Enabled() {\n\t\treturn errors.New(\"eval broker is enabled; eval broker must be paused to delete evals\")\n\t}\n\n\tif args.Filter != \"\" {\n\t\tcount, index, err := e.deleteEvalsByFilter(args)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\n\t\t// Update the index and return.\n\t\treply.Index = index\n\t\treply.Count = count\n\t\treturn nil\n\t}\n\n\t// Grab the state snapshot, so we can look up relevant eval information.\n\tserverStateSnapshot, err := e.srv.State().Snapshot()\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to lookup state snapshot: %v\", err)","sourceCodeStart":466,"sourceCodeEnd":502,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/eval_endpoint.go#L466-L502","documentation":"The Eval.Delete RPC refuses to delete evaluations while the eval broker is enabled. With the broker active, deleting evals directly would corrupt scheduler state (dequeueing evals that no longer exist), so Nomad requires the broker to be paused/disabled first.","triggerScenarios":"Calling Eval.Delete (RPC Eval.DeleteEval) with evals still flowing through the broker — i.e. any delete request made on a healthy, non-paused cluster, whether by ID or filter.","commonSituations":"Operators attempting cleanup of stuck/historical evals via 'nomad eval delete' or the API on an OSS cluster; automation scripts purging evals without pausing the broker first; debugging broker backlogs by mass deletion.","solutions":["Pause the eval broker (agent config / supported enterprise tooling) before deleting evals, then re-enable it","Use 'nomad eval purge' only in disaster-recovery scenarios as documented, with the broker disabled","Instead of deleting evals, cancel or let them complete; use 'nomad eval list' to triage","If doing state surgery, stop/safely pause scheduling, disable the broker, delete, then re-enable"],"exampleFix":"// before\nnomad eval delete -filter 'Status == \"blocked\"'\n// error: eval broker must be paused\n// after: pause broker via operator endpoint, then delete\ncurl -X POST $NOMAD/v1/operator/scheduler/pause-broker?enabled=true -H \"X-Nomad-Token: $TOKEN\"\nnomad eval delete -filter 'Status == \"blocked\"'","handlingStrategy":"validation","validationCode":"// shell: only attempt eval deletion when the broker is paused\nnomad operator scheduler get-config -json | jq -e '.PauseEvalBroker == true' \\\n  || { echo 'pause the eval broker first'; exit 1; }","typeGuard":null,"tryCatchPattern":"// shell\nif ! nomad eval delete \"$ID\" 2>&1 | grep -q 'eval broker is enabled'; then\n  echo \"deleted $ID\"\nelse\n  echo \"pause broker first\"\nfi","preventionTips":["Document that eval deletion requires pausing the broker (operator scheduler config)","Prefer eval cancel/let-GC over manual deletion","Perform broker-paused maintenance windows for eval cleanup","Re-enable the broker immediately after cleanup to avoid scheduler stalls"],"tags":["nomad","eval","scheduler","operator"],"backgroundTag":"eval-broker-not-paused","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}