hashicorp/nomad · critical

client.consul: unable to query Consul datacenters

Error message

client.consul: unable to query Consul datacenters: %v

What it means

`consulDiscoveryImpl`, the Nomad client's Consul-based server discovery path, queries the Consul catalog for datacenters via consulCatalog.Datacenters(). When that query fails (Consul unreachable, ACL denied, agent error), the error is wrapped as "client.consul: unable to query Consul datacenters: %v". Server discovery cannot proceed without the DC list, so this blocks the client from finding Nomad servers.

Solutions

1. Verify the Consul agent is healthy and reachable from the Nomad client (curl the Consul /v1/agent/health endpoint)
2. Check Nomad's consul.address/block and fix misconfiguration
3. Grant the Consul ACL token catalog read permissions (node:read, service:read nomad)
4. Inspect the wrapped %v cause in the message — connection refused vs ACL denied require different fixes

Example fix

// nomad.hcl
// before
consul {
  address = "consul.internal:8500" // unreachable
}
// after
consul {
  address = "127.0.0.1:8500"
  token   = "<acl-token-with-catalog-read>"
}

// pre-check Consul reachability before Nomad client start
resp, err := http.Get("http://127.0.0.1:8500/v1/status/leader")
if err != nil || resp.StatusCode != 200 {
    log.Fatal("Consul agent unreachable from Nomad client")
}

err := client.consulDiscoveryImpl()
if err != nil && strings.Contains(err.Error(), "unable to query Consul datacenters") {
    // inspect wrapped cause: connection refused vs ACL denied
    time.Backoff(retryWithJitter)
}

When it happens

Trigger: Client startup or heartbeat-driven server rediscovery while the local Consul agent is down, the HTTP address is misconfigured, or the Consul ACL token lacks catalog read permissions (node listing).

Common situations: Consul agent not running on the Nomad client node; incorrect consul.address in Nomad config; Consul ACL token missing `service:read`/`node:list` on catalog; network partition between Nomad client and Consul.

Related errors

• error retrieving supported Envoy versions from Consul
• failed to retrieve checks from consul
• failed to retrieve services from consul
• no Nomad Servers advertising service
• unable to query service %+q from Consul datacenter %+q

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/12c74e19b9ba6c0b. Report an issue: GitHub.

Appendix: source

Thrown at client/client.go:3129

func (c *Client) consulDiscovery() {
	for {
		select {
		case <-c.triggerDiscoveryCh:
			if err := c.consulDiscoveryImpl(); err != nil {
				c.logger.Error("error discovering nomad servers", "error", err)
			}
		case <-c.shutdownCh:
			return
		}
	}
}

func (c *Client) consulDiscoveryImpl() error {
	consulLogger := c.logger.Named("consul")

	dcs, err := c.consulCatalog.Datacenters()
	if err != nil {
		return fmt.Errorf("client.consul: unable to query Consul datacenters: %v", err)
	}
	if len(dcs) > 2 {
		// Query the local DC first, then shuffle the
		// remaining DCs.  Future heartbeats will cause Nomad
		// Clients to fixate on their local datacenter so
		// it's okay to talk with remote DCs.  If the no
		// Nomad servers are available within
		// datacenterQueryLimit, the next heartbeat will pick
		// a new set of servers so it's okay.
		shuffleStrings(dcs[1:])
		dcs = dcs[0:min(len(dcs), datacenterQueryLimit)]
	}

	serviceName := c.GetConfig().GetDefaultConsul().ServerServiceName
	var mErr multierror.Error
	var nomadServers servers.Servers
	consulLogger.Debug("bootstrap contacting Consul DCs", "consul_dcs", dcs)
DISCOLOOP:

View on GitHub (pinned to 482b49bf1a)