{"record":{"id":"12c74e19b9ba6c0b","repo":"hashicorp/nomad","slug":"client-consul-unable-to-query-consul-datacenters","errorCode":null,"errorMessage":"client.consul: unable to query Consul datacenters: %v","messagePattern":"client\\.consul: unable to query Consul datacenters: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"client/client.go","lineNumber":3129,"sourceCode":"func (c *Client) consulDiscovery() {\n\tfor {\n\t\tselect {\n\t\tcase <-c.triggerDiscoveryCh:\n\t\t\tif err := c.consulDiscoveryImpl(); err != nil {\n\t\t\t\tc.logger.Error(\"error discovering nomad servers\", \"error\", err)\n\t\t\t}\n\t\tcase <-c.shutdownCh:\n\t\t\treturn\n\t\t}\n\t}\n}\n\nfunc (c *Client) consulDiscoveryImpl() error {\n\tconsulLogger := c.logger.Named(\"consul\")\n\n\tdcs, err := c.consulCatalog.Datacenters()\n\tif err != nil {\n\t\treturn fmt.Errorf(\"client.consul: unable to query Consul datacenters: %v\", err)\n\t}\n\tif len(dcs) > 2 {\n\t\t// Query the local DC first, then shuffle the\n\t\t// remaining DCs.  Future heartbeats will cause Nomad\n\t\t// Clients to fixate on their local datacenter so\n\t\t// it's okay to talk with remote DCs.  If the no\n\t\t// Nomad servers are available within\n\t\t// datacenterQueryLimit, the next heartbeat will pick\n\t\t// a new set of servers so it's okay.\n\t\tshuffleStrings(dcs[1:])\n\t\tdcs = dcs[0:min(len(dcs), datacenterQueryLimit)]\n\t}\n\n\tserviceName := c.GetConfig().GetDefaultConsul().ServerServiceName\n\tvar mErr multierror.Error\n\tvar nomadServers servers.Servers\n\tconsulLogger.Debug(\"bootstrap contacting Consul DCs\", \"consul_dcs\", dcs)\nDISCOLOOP:","sourceCodeStart":3111,"sourceCodeEnd":3147,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/client.go#L3111-L3147","documentation":"`consulDiscoveryImpl`, the Nomad client's Consul-based server discovery path, queries the Consul catalog for datacenters via consulCatalog.Datacenters(). When that query fails (Consul unreachable, ACL denied, agent error), the error is wrapped as \"client.consul: unable to query Consul datacenters: %v\". Server discovery cannot proceed without the DC list, so this blocks the client from finding Nomad servers.","triggerScenarios":"Client startup or heartbeat-driven server rediscovery while the local Consul agent is down, the HTTP address is misconfigured, or the Consul ACL token lacks catalog read permissions (node listing).","commonSituations":"Consul agent not running on the Nomad client node; incorrect consul.address in Nomad config; Consul ACL token missing `service:read`/`node:list` on catalog; network partition between Nomad client and Consul.","solutions":["Verify the Consul agent is healthy and reachable from the Nomad client (curl the Consul /v1/agent/health endpoint)","Check Nomad's consul.address/block and fix misconfiguration","Grant the Consul ACL token catalog read permissions (node:read, service:read nomad)","Inspect the wrapped %v cause in the message — connection refused vs ACL denied require different fixes"],"exampleFix":"// nomad.hcl\n// before\nconsul {\n  address = \"consul.internal:8500\" // unreachable\n}\n// after\nconsul {\n  address = \"127.0.0.1:8500\"\n  token   = \"<acl-token-with-catalog-read>\"\n}","handlingStrategy":"retry","validationCode":"// pre-check Consul reachability before Nomad client start\nresp, err := http.Get(\"http://127.0.0.1:8500/v1/status/leader\")\nif err != nil || resp.StatusCode != 200 {\n    log.Fatal(\"Consul agent unreachable from Nomad client\")\n}","typeGuard":null,"tryCatchPattern":"err := client.consulDiscoveryImpl()\nif err != nil && strings.Contains(err.Error(), \"unable to query Consul datacenters\") {\n    // inspect wrapped cause: connection refused vs ACL denied\n    time.Backoff(retryWithJitter)\n}","preventionTips":["Run a local Consul agent on every Nomad client node","Verify the consul.address config and ACL token catalog permissions","Monitor Consul health from the same network path Nomad uses","Read the wrapped %v cause to distinguish connectivity vs ACL problems"],"tags":["nomad","consul","service-discovery","network"],"backgroundTag":"consul-query-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}