hashicorp/nomad · warning

failed to query service %q in Consul datacenter %q: %v

Error message

failed to query service %q in Consul datacenter %q: %v

What it means

Also inside setupBootstrapHandler's fallback path: after listing datacenters, the server queries Consul for the "nomad" service (serf tag) in each DC via s.consulCatalog.Service. Failures are accumulated per datacenter into a multierror and returned, then the poll retries later. One bad DC does not stop the loop; the aggregate error reports all failing DCs.

Source

Thrown at nomad/server.go:1071

			shuffleStrings(dcs[1:])
			dcs = dcs[0:min(len(dcs), datacenterQueryLimit)]
		}

		nomadServerServiceName := s.config.GetDefaultConsul().ServerServiceName
		var mErr multierror.Error
		const defaultMaxNumNomadServers = 8
		nomadServerServices := make([]string, 0, defaultMaxNumNomadServers)
		localNode := s.serf.Memberlist().LocalNode()
		for _, dc := range dcs {
			consulOpts := &consulapi.QueryOptions{
				AllowStale: true,
				Datacenter: dc,
				Near:       "_agent",
				WaitTime:   consul.DefaultQueryWaitDuration,
			}
			consulServices, _, err := s.consulCatalog.Service(nomadServerServiceName, consul.ServiceTagSerf, consulOpts)
			if err != nil {
				err := fmt.Errorf("failed to query service %q in Consul datacenter %q: %v", nomadServerServiceName, dc, err)
				s.logger.Warn("failed to query Nomad service in Consul datacenter", "service_name", nomadServerServiceName, "dc", dc, "error", err)
				mErr.Errors = append(mErr.Errors, err)
				continue
			}

			for _, cs := range consulServices {
				port := strconv.FormatInt(int64(cs.ServicePort), 10)
				addr := cs.ServiceAddress
				if addr == "" {
					addr = cs.Address
				}
				if localNode.Addr.String() == addr && int(localNode.Port) == cs.ServicePort {
					continue
				}
				serverAddr := net.JoinHostPort(addr, port)
				nomadServerServices = append(nomadServerServices, serverAddr)
			}
		}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Fix Consul connectivity/health in the DC(s) named in the error.
2. Verify the Consul ACL token can read the nomad service (service:read).
3. Check WAN gossip/connectivity between datacenters and Consul query timeouts.
4. Restore Nomad quorum so the Consul bootstrap fallback stops being used.

Example fix

// before
# ACL policy missing service read
node "" { policy = "read" }
// after
service "nomad" { policy = "read" }
node "" { policy = "read" }

// preflight per DC
for _, dc := range dcs {
    _, _, err := consulClient.Catalog().Service("nomad", consul.ServiceTagSerf, &api.QueryOptions{Datacenter: dc})
    if err != nil { log.Printf("dc %s not queryable: %v", dc, err) }
}

// handler retries automatically; in automation treat this as transient
if strings.Contains(errLog, "failed to query service") {
    waitForConsulHealth(dcName, 5*time.Minute)
}

When it happens

Trigger: Quorum lost (fallback active), and consulCatalog.Service() errors for a DC — Consul API down, ACL token missing service read, query timeout against remote DCs (WAN issues), TLS problems.

Common situations: Multi-DC deployments where a remote DC's Consul is unreachable over the WAN, stale ACL tokens after rotation, Consul servers in one region down.

Related errors

• failed to retrieve services from consul: %w
• no one-time token returned
• no ACL token returned
• errMissingACLRoleID
• errMissingACLAuthMethodName

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/496e5fdb0c8426fe. Report an issue: GitHub.