hashicorp/nomad · warning

server.nomad: unable to query Consul datacenters: %v

Error message

server.nomad: unable to query Consul datacenters: %v

What it means

Inside the bootstrap handler registered by setupConsulSyncer, when the Nomad quorum is unreachable the server falls back to discovering peers via Consul's catalog. If s.consulCatalog.Datacenters() fails, the poll returns this error and backs off by resetting the peers timer. The syncer retries periodically, so this error can appear repeatedly while Consul is degraded.

Source

Thrown at nomad/server.go:1045

			}

			// The necessary number of Nomad Servers required for
			// quorum has been reached, we do not need to poll
			// Consul.  Let the normal timeout-based strategy
			// take over.
			if raftPeers >= bootstrapExpect {
				peersTimeout.Reset(peersPollInterval + helper.RandomStagger(peersPollInterval/peersPollJitterFactor))
				return nil
			}
		}
		consulQueryCount++

		s.logger.Debug("lost contact with Nomad quorum, falling back to Consul for server list")

		dcs, err := s.consulCatalog.Datacenters()
		if err != nil {
			peersTimeout.Reset(peersPollInterval + helper.RandomStagger(peersPollInterval/peersPollJitterFactor))
			return fmt.Errorf("server.nomad: unable to query Consul datacenters: %v", err)
		}
		if len(dcs) > 2 {
			// Query the local DC first, then shuffle the
			// remaining DCs.  If additional calls to bootstrapFn
			// are necessary, this Nomad Server will eventually
			// walk all datacenter until it finds enough hosts to
			// form a quorum.
			shuffleStrings(dcs[1:])
			dcs = dcs[0:min(len(dcs), datacenterQueryLimit)]
		}

		nomadServerServiceName := s.config.GetDefaultConsul().ServerServiceName
		var mErr multierror.Error
		const defaultMaxNumNomadServers = 8
		nomadServerServices := make([]string, 0, defaultMaxNumNomadServers)
		localNode := s.serf.Memberlist().LocalNode()
		for _, dc := range dcs {
			consulOpts := &consulapi.QueryOptions{

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Restore Nomad quorum (start/repair enough servers) so the Consul fallback isn't needed.
2. Verify the Consul agent address/health the Nomad server points at.
3. Check the Consul ACL token has `service:read`/catalog list permissions.
4. Fix Consul TLS (ca_file/cert/key) if certificates were rotated.

Example fix

// before
consul {
  token = "token-without-catalog-read"
}
// after
consul {
  token = "<token granting service:read on nomad>"
}

// preflight: catalog is queryable with the configured token
resp, err := consulClient.Catalog().Datacenters()
// treat err != nil or empty resp as blocking for server startup

if err := srv.Reload(...); strings.Contains(logs, "unable to query Consul datacenters") {
    // transient: the bootstrap handler retries automatically with backoff
    time.Sleep(pollInterval)
    // re-check quorum status via /v1/status/peers
}

When it happens

Trigger: Cluster lost quorum (fallback path active) AND the Consul catalog query fails — Consul agent down, bad consul address, ACL token lacking catalog read permissions, TLS mismatch with Consul.

Common situations: Datacenter outage where both Nomad and Consul are impaired; Consul ACL policy tightened to remove service:catalog read; misconfigured consul.tls blocks after cert rotation.

Related errors

• %w: %v; see: <https://developer.hashicorp.com/nomad/s/envoy-
• missing %q
• no one-time token returned
• no ACL token returned
• errMissingACLRoleID

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/d17bfc5f34b8551e. Report an issue: GitHub.