{"record":{"id":"e2bed141a09d99e0","repo":"hashicorp/nomad","slug":"eval-broker-is-enabled-eval-broker-must-be-paused-e2bed1","errorCode":null,"errorMessage":"eval broker is enabled; eval broker must be paused to delete evals","messagePattern":"eval broker is enabled; eval broker must be paused to delete evals","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":3604,"sourceCode":"\tif err := txn.Insert(\"index\", &IndexEntry{\"evals\", index}); err != nil {\n\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t}\n\treturn nil\n}\n\n// DeleteEvalsByFilter is used to delete all evals that are both safe to delete\n// and match a filter.\nfunc (s *StateStore) DeleteEvalsByFilter(index uint64, filterExpr string, pageToken string, perPage int32) error {\n\ttxn := s.db.WriteTxn(index)\n\tdefer txn.Abort()\n\n\t// These are always user-initiated, so ensure the eval broker is paused.\n\t_, schedConfig, err := s.schedulerConfigTxn(txn)\n\tif err != nil {\n\t\treturn err\n\t}\n\tif schedConfig == nil || !schedConfig.PauseEvalBroker {\n\t\treturn errors.New(\"eval broker is enabled; eval broker must be paused to delete evals\")\n\t}\n\n\tfilter, err := bexpr.CreateEvaluator(filterExpr)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\titer, err := s.Evals(nil, SortDefault)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to lookup evals: %v\", err)\n\t}\n\n\t// Note: Paginator imports this package for testing so we can't just use\n\t// Paginator\n\tpageCount := int32(0)\n\n\tfor {\n\t\tif pageCount >= perPage {","sourceCodeStart":3586,"sourceCodeEnd":3622,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L3586-L3622","documentation":"DeleteEvals is a destructive, user-initiated operation that must run while the eval broker is paused so in-flight scheduling state is frozen. The state store validates scheduler config inside the write txn and rejects the delete if PauseEvalBroker is not set.","triggerScenarios":"Calling StateStore.DeleteEvals (exposed via `nomad system gc`-style eval deletion or the evals DELETE API) while the scheduler config has PauseEvalBroker=false or is nil.","commonSituations":"Operators running bulk eval cleanup during incident triage without first pausing the broker; automation that deletes evals as part of garbage collection; mistaken assumption deletion can happen live.","solutions":["Pause the eval broker first: `nomad operator scheduler set-config -pause-eval-broker=true`, then retry the deletion","Delete evals again after cleanup and unpause with -pause-eval-broker=false","If using the HTTP API, PUT /v1/operator/scheduler/configuration with PauseEvalBroker:true before deleting"],"exampleFix":"// before\nnomad system gc  # deletes evals with broker running -> error\n// after\nnomad operator scheduler set-config -pause-eval-broker=true\nnomad system gc\nnomad operator scheduler set-config -pause-eval-broker=false","handlingStrategy":"validation","validationCode":"cfg, _, _ := client.Operator().SchedulerGetConfiguration(nil)\nif cfg.SchedulerConfig == nil || !cfg.SchedulerConfig.PauseEvalBroker {\n    _, _, err := client.Operator().SchedulerSetConfiguration(\n        (&api.SchedulerConfiguration{PauseEvalBroker: true}).Canonicalize(), nil, nil)\n    if err != nil { return err }\n    defer unpauseEvalBroker(client)\n}\n// proceed with eval deletion","typeGuard":"func brokerPaused(c *api.SchedulerConfiguration) bool {\n    return c != nil && c.PauseEvalBroker\n}","tryCatchPattern":"if err := deleteEvals(client, filter); err != nil && strings.Contains(err.Error(), \"eval broker must be paused\") {\n    pauseBroker(client); defer unpauseBroker(client)\n    err = deleteEvals(client, filter)\n}","preventionTips":["Always pause the eval broker before maintenance deletions","Unpause immediately after maintenance completes","Script pause/delete/unpause as one atomic runbook"],"tags":["nomad","state-store","evals","maintenance","scheduler"],"backgroundTag":"eval-broker-must-be-paused","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}