{"record":{"id":"f71916196957fcb2","repo":"hashicorp/nomad","slug":"all-servers-must-be-running-version-v-or-later-to","errorCode":null,"errorMessage":"all servers must be running version %v or later to delete evals by filter","messagePattern":"all servers must be running version (.+?) or later to delete evals by filter","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/eval_endpoint.go","lineNumber":470,"sourceCode":"\t\treturn err\n\t}\n\te.srv.MeasureRPCRate(\"eval\", structs.RateMetricWrite, args)\n\tif authErr != nil {\n\t\treturn structs.ErrPermissionDenied\n\t}\n\tdefer metrics.MeasureSince([]string{\"nomad\", \"eval\", \"delete\"}, time.Now())\n\n\t// This RPC endpoint is very destructive and alters Nomad's core state,\n\t// meaning only those with management tokens can call it.\n\tif aclObj, err := e.srv.ResolveACL(args); err != nil {\n\t\treturn err\n\t} else if !aclObj.IsManagement() {\n\t\treturn structs.ErrPermissionDenied\n\t}\n\n\tif args.Filter != \"\" && !e.srv.peersCache.ServersMeetMinimumVersion(\n\t\te.srv.Region(), minVersionEvalDeleteByFilter, true) {\n\t\treturn fmt.Errorf(\n\t\t\t\"all servers must be running version %v or later to delete evals by filter\",\n\t\t\tminVersionEvalDeleteByFilter)\n\t}\n\tif args.Filter != \"\" && len(args.EvalIDs) > 0 {\n\t\treturn fmt.Errorf(\"evals cannot be deleted by both ID and filter\")\n\t}\n\tif args.Filter == \"\" && len(args.EvalIDs) == 0 {\n\t\treturn fmt.Errorf(\"evals must be deleted by either ID or filter\")\n\t}\n\n\t// The eval broker must be disabled otherwise Nomad's state will likely get\n\t// wild in a very un-fun way.\n\tif e.srv.evalBroker.Enabled() {\n\t\treturn errors.New(\"eval broker is enabled; eval broker must be paused to delete evals\")\n\t}\n\n\tif args.Filter != \"\" {\n\t\tcount, index, err := e.deleteEvalsByFilter(args)","sourceCodeStart":452,"sourceCodeEnd":488,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/eval_endpoint.go#L452-L488","documentation":"Returned by Evaluations.Delete when a filter-based delete is requested but not every server in the region meets the minimum version that supports eval delete-by-filter. The operation requires cluster-wide support because the filter is evaluated consistently across servers.","triggerScenarios":"Calling the eval delete API with a non-empty args.Filter while e.srv.peersCache.ServersMeetMinimumVersion reports some server in the region is older than minVersionEvalDeleteByFilter.","commonSituations":"Rolling upgrades where one or more servers still run an older Nomad version; forgetting to upgrade all client servers (not just leaders) before using filter-based eval cleanup APIs.","solutions":["Upgrade all servers in the region to at least minVersionEvalDeleteByFilter","Delete evals by explicit IDs instead of a filter until the cluster is fully upgraded","Verify server versions (nomad server members / /v1/agent/members) before retrying"],"exampleFix":"// before\nreq := &api.EvalDeleteRequest{Filter: \"Status == complete\"}\n// after\nreq := &api.EvalDeleteRequest{EvalIDs: []string{evalID}} // until all servers upgraded","handlingStrategy":"validation","validationCode":"members, _, err := client.Agent().Members()\nif err != nil { return err }\nfor _, m := range members.Members {\n    if version.Compare(m.Tags[\"build\"], \"0.11.5\") < 0 { return fmt.Errorf(\"server %s too old\", m.Name) }\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep all servers on a uniform version before using filter APIs","Check nomad server members' build tags before issuing filter deletes","Fall back to ID-based deletes during rolling upgrades"],"tags":["nomad","rpc","version-compatibility","cluster-upgrade"],"backgroundTag":"cluster-version-mismatch","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}