hashicorp/nomad · error

unable to query service %+q from Consul datacenter %+q: %v

Error message

unable to query service %+q from Consul datacenter %+q: %v

What it means

Within Consul server discovery, the client queries the `nomad` RPC service in each discovered datacenter. If the catalog Service() call fails for a particular DC, the error is appended to a multi-error as "unable to query service %+q from Consul datacenter %+q: %v" and discovery continues with other DCs. It indicates a per-datacenter Consul query failure rather than total discovery failure.

Source

Thrown at client/client.go:3157

		shuffleStrings(dcs[1:])
		dcs = dcs[0:min(len(dcs), datacenterQueryLimit)]
	}

	serviceName := c.GetConfig().GetDefaultConsul().ServerServiceName
	var mErr multierror.Error
	var nomadServers servers.Servers
	consulLogger.Debug("bootstrap contacting Consul DCs", "consul_dcs", dcs)
DISCOLOOP:
	for _, dc := range dcs {
		consulOpts := &consulapi.QueryOptions{
			AllowStale: true,
			Datacenter: dc,
			Near:       "_agent",
			WaitTime:   consul.DefaultQueryWaitDuration,
		}
		consulServices, _, err := c.consulCatalog.Service(serviceName, consul.ServiceTagRPC, consulOpts)
		if err != nil {
			mErr.Errors = append(mErr.Errors, fmt.Errorf("unable to query service %+q from Consul datacenter %+q: %v", serviceName, dc, err))
			continue
		}

		for _, s := range consulServices {
			port := strconv.Itoa(s.ServicePort)
			addrstr := s.ServiceAddress
			if addrstr == "" {
				addrstr = s.Address
			}
			addr, err := net.ResolveTCPAddr("tcp", net.JoinHostPort(addrstr, port))
			if err != nil {
				mErr.Errors = append(mErr.Errors, err)
				continue
			}

			srv := &servers.Server{Addr: addr}
			nomadServers = append(nomadServers, srv)
		}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the health of the Consul agent in the named datacenter and its WAN federation links
2. Verify the ACL token has service:read on the `nomad` service in that DC
3. Inspect the wrapped cause; fix connectivity (firewall/DNS) between the client and that DC's Consul servers
4. If the DC is decommissioned, remove it from Consul so discovery stops querying it

Example fix

// before: token limited to one DC
// acl token: service "nomad" { policy = "read" } only in dc1
// after: grant read across federated DCs
// acl token: service "nomad" { policy = "read" } replicated to all DCs

// verify nomad service exists in Consul before discovery
cs, _, err := consul.Catalog().Service("nomad", "", nil)
if err != nil || len(cs) == 0 {
    log.Printf("nomad service not queryable in Consul: %v", err)
}

if err := discover(); err != nil {
    var mErr multierror.Error
    if errors.As(err, &mErr) {
        for _, e := range mErr.Errors {
            // handle per-DC failures; retry healthy DCs
        }
    }
}

When it happens

Trigger: Consul catalog Service lookup for the `nomad` RPC service failing in one DC — Consul agent in that DC down/unreachable, ACL token lacking service read on `nomad`, or cross-DC catalog query timing out.

Common situations: Multi-datacenter setups where one remote DC's Consul is degraded; ACLs that restrict service discovery to the local DC; WAN-federation issues causing cross-DC catalog queries to fail.

Related errors

• client.consul: unable to query Consul datacenters: %v
• no Nomad Servers advertising service %q in Consul datacenter
• failed to retrieve services from consul: %w
• error creating bootstrap configuration for Connect proxy sid
• client stopped and may not longer create config entries

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/8a25f85a9356d265. Report an issue: GitHub.