{"record":{"id":"d17bfc5f34b8551e","repo":"hashicorp/nomad","slug":"server-nomad-unable-to-query-consul-datacenters","errorCode":null,"errorMessage":"server.nomad: unable to query Consul datacenters: %v","messagePattern":"server\\.nomad: unable to query Consul datacenters: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"nomad/server.go","lineNumber":1045,"sourceCode":"\t\t\t}\n\n\t\t\t// The necessary number of Nomad Servers required for\n\t\t\t// quorum has been reached, we do not need to poll\n\t\t\t// Consul.  Let the normal timeout-based strategy\n\t\t\t// take over.\n\t\t\tif raftPeers >= bootstrapExpect {\n\t\t\t\tpeersTimeout.Reset(peersPollInterval + helper.RandomStagger(peersPollInterval/peersPollJitterFactor))\n\t\t\t\treturn nil\n\t\t\t}\n\t\t}\n\t\tconsulQueryCount++\n\n\t\ts.logger.Debug(\"lost contact with Nomad quorum, falling back to Consul for server list\")\n\n\t\tdcs, err := s.consulCatalog.Datacenters()\n\t\tif err != nil {\n\t\t\tpeersTimeout.Reset(peersPollInterval + helper.RandomStagger(peersPollInterval/peersPollJitterFactor))\n\t\t\treturn fmt.Errorf(\"server.nomad: unable to query Consul datacenters: %v\", err)\n\t\t}\n\t\tif len(dcs) > 2 {\n\t\t\t// Query the local DC first, then shuffle the\n\t\t\t// remaining DCs.  If additional calls to bootstrapFn\n\t\t\t// are necessary, this Nomad Server will eventually\n\t\t\t// walk all datacenter until it finds enough hosts to\n\t\t\t// form a quorum.\n\t\t\tshuffleStrings(dcs[1:])\n\t\t\tdcs = dcs[0:min(len(dcs), datacenterQueryLimit)]\n\t\t}\n\n\t\tnomadServerServiceName := s.config.GetDefaultConsul().ServerServiceName\n\t\tvar mErr multierror.Error\n\t\tconst defaultMaxNumNomadServers = 8\n\t\tnomadServerServices := make([]string, 0, defaultMaxNumNomadServers)\n\t\tlocalNode := s.serf.Memberlist().LocalNode()\n\t\tfor _, dc := range dcs {\n\t\t\tconsulOpts := &consulapi.QueryOptions{","sourceCodeStart":1027,"sourceCodeEnd":1063,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/server.go#L1027-L1063","documentation":"Inside the bootstrap handler registered by setupConsulSyncer, when the Nomad quorum is unreachable the server falls back to discovering peers via Consul's catalog. If s.consulCatalog.Datacenters() fails, the poll returns this error and backs off by resetting the peers timer. The syncer retries periodically, so this error can appear repeatedly while Consul is degraded.","triggerScenarios":"Cluster lost quorum (fallback path active) AND the Consul catalog query fails — Consul agent down, bad consul address, ACL token lacking catalog read permissions, TLS mismatch with Consul.","commonSituations":"Datacenter outage where both Nomad and Consul are impaired; Consul ACL policy tightened to remove service:catalog read; misconfigured consul.tls blocks after cert rotation.","solutions":["Restore Nomad quorum (start/repair enough servers) so the Consul fallback isn't needed.","Verify the Consul agent address/health the Nomad server points at.","Check the Consul ACL token has `service:read`/catalog list permissions.","Fix Consul TLS (ca_file/cert/key) if certificates were rotated."],"exampleFix":"// before\nconsul {\n  token = \"token-without-catalog-read\"\n}\n// after\nconsul {\n  token = \"<token granting service:read on nomad>\"\n}","handlingStrategy":"retry","validationCode":"// preflight: catalog is queryable with the configured token\nresp, err := consulClient.Catalog().Datacenters()\n// treat err != nil or empty resp as blocking for server startup","typeGuard":null,"tryCatchPattern":"if err := srv.Reload(...); strings.Contains(logs, \"unable to query Consul datacenters\") {\n    // transient: the bootstrap handler retries automatically with backoff\n    time.Sleep(pollInterval)\n    // re-check quorum status via /v1/status/peers\n}","preventionTips":["Monitor Consul agent health alongside Nomad.","Grant the Nomad Consul token catalog/service read ACLs.","Keep a Consul agent co-located with Nomad servers."],"tags":["consul","discovery","quorum","retry"],"backgroundTag":"consul-catalog-query-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}