{"record":{"id":"58ec4f3dd203574d","repo":"hashicorp/nomad","slug":"failed-to-retrieve-services-from-consul-w","errorCode":null,"errorMessage":"failed to retrieve services from consul: %w","messagePattern":"failed to retrieve services from consul: %w","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"command/agent/consul/service_client.go","lineNumber":1793,"sourceCode":"\t// Get the list of all namespaces created so we can iterate them.\n\tnamespaces, err := c.namespacesClient.List()\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to retrieve namespaces from consul: %w\", err)\n\t}\n\n\tservices := make(map[string]*api.AgentService)\n\tchecks := make(map[string]*api.AgentCheck)\n\n\t// Query the services and checks to populate the allocation registrations.\n\t// Note: these queries have to use the Nomad agent's own Consul token\n\tfor _, namespace := range namespaces {\n\t\tqo := &api.QueryOptions{\n\t\t\tNamespace: normalizeNamespace(namespace),\n\t\t}\n\n\t\tnsServices, err := c.agentAPI.ServicesWithFilterOpts(\"\", qo)\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"failed to retrieve services from consul: %w\", err)\n\t\t}\n\t\tmaps.Copy(services, nsServices)\n\n\t\tnsChecks, err := c.agentAPI.ChecksWithFilterOpts(\"\", qo)\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"failed to retrieve checks from consul: %w\", err)\n\t\t}\n\t\tmaps.Copy(checks, nsChecks)\n\t}\n\n\t// Populate the object\n\tfor _, treg := range reg.Tasks {\n\t\tfor serviceID, sreg := range treg.Services {\n\t\t\tsreg.Service = services[serviceID]\n\t\t\tfor checkID := range sreg.CheckIDs {\n\t\t\t\tif check, ok := checks[checkID]; ok {\n\t\t\t\t\tsreg.Checks = append(sreg.Checks, check)\n\t\t\t\t}","sourceCodeStart":1775,"sourceCodeEnd":1811,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/command/agent/consul/service_client.go#L1775-L1811","documentation":"While enumerating every Consul namespace to gather all agent services, Nomad calls agentAPI.ServicesWithFilterOpts per namespace. Any failure from the Consul agent API (ACL denial on service:read, agent unreachable, malformed response) is wrapped in this message and AllocRegistrations returns nil, preventing reconciliation of service registrations.","triggerScenarios":"AllocRegistrations iterates namespaces; for each, c.agentAPI.ServicesWithFilterOpts(\"\", qo) with a namespace-scoped QueryOptions errors — typically ACL token missing service:read in that namespace or Consul agent connection failure.","commonSituations":"Wildcard/multi-namespace setups where the Nomad token lacks service:read in one namespace; Consul agent restarted or listening on a different address; transient network blip during reconciliation causing spurious errors.","solutions":["Grant the Nomad Consul token service:read (service:write) in every namespace Nomad manages, or restrict consul namespaces config to ones it can read","Verify the Consul agent HTTP address/port in the Nomad consul stanza and test with `consul catalog services` using the same token","Retry if transient — reconciliation reruns periodically; a persistent error indicates ACL/address issues","Check the wrapped %w error string for 'Permission denied' (ACL) vs connection errors"],"exampleFix":"// before (Consul ACL policy)\nnamespace \"nomad\" {\n  // no service permissions\n}\n// after\nnamespace \"nomad\" {\n  service_prefix \"\" {\n    policy = \"write\"\n  }\n  key_prefix \"\" {\n    policy = \"read\"\n  }\n}","handlingStrategy":"retry","validationCode":"// Pre-flight the same API call the client makes, using the Nomad token\n_, _, err := consulClient.Agent().ServicesWithFilterOpts(\"\", &api.QueryOptions{Namespace: ns})\nif err != nil {\n  log.Printf(\"token cannot read services in ns %q: %v\", ns, err)\n}","typeGuard":null,"tryCatchPattern":"regs, err := client.AllocRegistrations(allocID)\nif err != nil {\n  if strings.Contains(err.Error(), \"failed to retrieve services\") {\n    // transient or ACL: schedule a retry / alert on repeated failure\n  }\n}","preventionTips":["Ensure service:write (implies read) ACLs in every managed namespace","Keep the Nomad-to-Consul HTTP address stable across agent restarts","Treat single occurrences as transient; alert on persistence","Test token permissions with `consul catalog services -namespace=<ns>`"],"tags":["consul","nomad","acl","service-discovery"],"backgroundTag":"consul-api-request-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}