{"record":{"id":"8a25f85a9356d265","repo":"hashicorp/nomad","slug":"unable-to-query-service-q-from-consul-datacenter","errorCode":null,"errorMessage":"unable to query service %+q from Consul datacenter %+q: %v","messagePattern":"unable to query service %\\+q from Consul datacenter %\\+q: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/client.go","lineNumber":3157,"sourceCode":"\t\tshuffleStrings(dcs[1:])\n\t\tdcs = dcs[0:min(len(dcs), datacenterQueryLimit)]\n\t}\n\n\tserviceName := c.GetConfig().GetDefaultConsul().ServerServiceName\n\tvar mErr multierror.Error\n\tvar nomadServers servers.Servers\n\tconsulLogger.Debug(\"bootstrap contacting Consul DCs\", \"consul_dcs\", dcs)\nDISCOLOOP:\n\tfor _, dc := range dcs {\n\t\tconsulOpts := &consulapi.QueryOptions{\n\t\t\tAllowStale: true,\n\t\t\tDatacenter: dc,\n\t\t\tNear:       \"_agent\",\n\t\t\tWaitTime:   consul.DefaultQueryWaitDuration,\n\t\t}\n\t\tconsulServices, _, err := c.consulCatalog.Service(serviceName, consul.ServiceTagRPC, consulOpts)\n\t\tif err != nil {\n\t\t\tmErr.Errors = append(mErr.Errors, fmt.Errorf(\"unable to query service %+q from Consul datacenter %+q: %v\", serviceName, dc, err))\n\t\t\tcontinue\n\t\t}\n\n\t\tfor _, s := range consulServices {\n\t\t\tport := strconv.Itoa(s.ServicePort)\n\t\t\taddrstr := s.ServiceAddress\n\t\t\tif addrstr == \"\" {\n\t\t\t\taddrstr = s.Address\n\t\t\t}\n\t\t\taddr, err := net.ResolveTCPAddr(\"tcp\", net.JoinHostPort(addrstr, port))\n\t\t\tif err != nil {\n\t\t\t\tmErr.Errors = append(mErr.Errors, err)\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\tsrv := &servers.Server{Addr: addr}\n\t\t\tnomadServers = append(nomadServers, srv)\n\t\t}","sourceCodeStart":3139,"sourceCodeEnd":3175,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/client.go#L3139-L3175","documentation":"Within Consul server discovery, the client queries the `nomad` RPC service in each discovered datacenter. If the catalog Service() call fails for a particular DC, the error is appended to a multi-error as \"unable to query service %+q from Consul datacenter %+q: %v\" and discovery continues with other DCs. It indicates a per-datacenter Consul query failure rather than total discovery failure.","triggerScenarios":"Consul catalog Service lookup for the `nomad` RPC service failing in one DC — Consul agent in that DC down/unreachable, ACL token lacking service read on `nomad`, or cross-DC catalog query timing out.","commonSituations":"Multi-datacenter setups where one remote DC's Consul is degraded; ACLs that restrict service discovery to the local DC; WAN-federation issues causing cross-DC catalog queries to fail.","solutions":["Check the health of the Consul agent in the named datacenter and its WAN federation links","Verify the ACL token has service:read on the `nomad` service in that DC","Inspect the wrapped cause; fix connectivity (firewall/DNS) between the client and that DC's Consul servers","If the DC is decommissioned, remove it from Consul so discovery stops querying it"],"exampleFix":"// before: token limited to one DC\n// acl token: service \"nomad\" { policy = \"read\" } only in dc1\n// after: grant read across federated DCs\n// acl token: service \"nomad\" { policy = \"read\" } replicated to all DCs","handlingStrategy":"retry","validationCode":"// verify nomad service exists in Consul before discovery\ncs, _, err := consul.Catalog().Service(\"nomad\", \"\", nil)\nif err != nil || len(cs) == 0 {\n    log.Printf(\"nomad service not queryable in Consul: %v\", err)\n}","typeGuard":null,"tryCatchPattern":"if err := discover(); err != nil {\n    var mErr multierror.Error\n    if errors.As(err, &mErr) {\n        for _, e := range mErr.Errors {\n            // handle per-DC failures; retry healthy DCs\n        }\n    }\n}","preventionTips":["Monitor Consul agent health in every federated datacenter","Ensure the ACL token has service:read on `nomad` in all DCs","Maintain WAN federation between DC Consuls","Remove decommissioned DCs from Consul"],"tags":["nomad","consul","multi-datacenter","service-discovery"],"backgroundTag":"consul-service-lookup-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}