hashicorp/nomad · critical

client.consul: unable to query Consul datacenters: %v

Error message

client.consul: unable to query Consul datacenters: %v

What it means

`consulDiscoveryImpl`, the Nomad client's Consul-based server discovery path, queries the Consul catalog for datacenters via consulCatalog.Datacenters(). When that query fails (Consul unreachable, ACL denied, agent error), the error is wrapped as "client.consul: unable to query Consul datacenters: %v". Server discovery cannot proceed without the DC list, so this blocks the client from finding Nomad servers.

Source

Thrown at client/client.go:3129

func (c *Client) consulDiscovery() {
	for {
		select {
		case <-c.triggerDiscoveryCh:
			if err := c.consulDiscoveryImpl(); err != nil {
				c.logger.Error("error discovering nomad servers", "error", err)
			}
		case <-c.shutdownCh:
			return
		}
	}
}

func (c *Client) consulDiscoveryImpl() error {
	consulLogger := c.logger.Named("consul")

	dcs, err := c.consulCatalog.Datacenters()
	if err != nil {
		return fmt.Errorf("client.consul: unable to query Consul datacenters: %v", err)
	}
	if len(dcs) > 2 {
		// Query the local DC first, then shuffle the
		// remaining DCs.  Future heartbeats will cause Nomad
		// Clients to fixate on their local datacenter so
		// it's okay to talk with remote DCs.  If the no
		// Nomad servers are available within
		// datacenterQueryLimit, the next heartbeat will pick
		// a new set of servers so it's okay.
		shuffleStrings(dcs[1:])
		dcs = dcs[0:min(len(dcs), datacenterQueryLimit)]
	}

	serviceName := c.GetConfig().GetDefaultConsul().ServerServiceName
	var mErr multierror.Error
	var nomadServers servers.Servers
	consulLogger.Debug("bootstrap contacting Consul DCs", "consul_dcs", dcs)
DISCOLOOP:

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify the Consul agent is healthy and reachable from the Nomad client (curl the Consul /v1/agent/health endpoint)
2. Check Nomad's consul.address/block and fix misconfiguration
3. Grant the Consul ACL token catalog read permissions (node:read, service:read nomad)
4. Inspect the wrapped %v cause in the message — connection refused vs ACL denied require different fixes

Example fix

// nomad.hcl
// before
consul {
  address = "consul.internal:8500" // unreachable
}
// after
consul {
  address = "127.0.0.1:8500"
  token   = "<acl-token-with-catalog-read>"
}

// pre-check Consul reachability before Nomad client start
resp, err := http.Get("http://127.0.0.1:8500/v1/status/leader")
if err != nil || resp.StatusCode != 200 {
    log.Fatal("Consul agent unreachable from Nomad client")
}

err := client.consulDiscoveryImpl()
if err != nil && strings.Contains(err.Error(), "unable to query Consul datacenters") {
    // inspect wrapped cause: connection refused vs ACL denied
    time.Backoff(retryWithJitter)
}

When it happens

Trigger: Client startup or heartbeat-driven server rediscovery while the local Consul agent is down, the HTTP address is misconfigured, or the Consul ACL token lacks catalog read permissions (node listing).

Common situations: Consul agent not running on the Nomad client node; incorrect consul.address in Nomad config; Consul ACL token missing `service:read`/`node:list` on catalog; network partition between Nomad client and Consul.

Related errors

• error retrieving supported Envoy versions from Consul: %w
• unable to query service %+q from Consul datacenter %+q: %v
• no Nomad Servers advertising service %q in Consul datacenter
• failed to retrieve services from consul: %w
• failed to retrieve checks from consul: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/12c74e19b9ba6c0b. Report an issue: GitHub.