{"record":{"id":"496e5fdb0c8426fe","repo":"hashicorp/nomad","slug":"failed-to-query-service-q-in-consul-datacenter-q","errorCode":null,"errorMessage":"failed to query service %q in Consul datacenter %q: %v","messagePattern":"failed to query service %q in Consul datacenter %q: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"nomad/server.go","lineNumber":1071,"sourceCode":"\t\t\tshuffleStrings(dcs[1:])\n\t\t\tdcs = dcs[0:min(len(dcs), datacenterQueryLimit)]\n\t\t}\n\n\t\tnomadServerServiceName := s.config.GetDefaultConsul().ServerServiceName\n\t\tvar mErr multierror.Error\n\t\tconst defaultMaxNumNomadServers = 8\n\t\tnomadServerServices := make([]string, 0, defaultMaxNumNomadServers)\n\t\tlocalNode := s.serf.Memberlist().LocalNode()\n\t\tfor _, dc := range dcs {\n\t\t\tconsulOpts := &consulapi.QueryOptions{\n\t\t\t\tAllowStale: true,\n\t\t\t\tDatacenter: dc,\n\t\t\t\tNear:       \"_agent\",\n\t\t\t\tWaitTime:   consul.DefaultQueryWaitDuration,\n\t\t\t}\n\t\t\tconsulServices, _, err := s.consulCatalog.Service(nomadServerServiceName, consul.ServiceTagSerf, consulOpts)\n\t\t\tif err != nil {\n\t\t\t\terr := fmt.Errorf(\"failed to query service %q in Consul datacenter %q: %v\", nomadServerServiceName, dc, err)\n\t\t\t\ts.logger.Warn(\"failed to query Nomad service in Consul datacenter\", \"service_name\", nomadServerServiceName, \"dc\", dc, \"error\", err)\n\t\t\t\tmErr.Errors = append(mErr.Errors, err)\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\tfor _, cs := range consulServices {\n\t\t\t\tport := strconv.FormatInt(int64(cs.ServicePort), 10)\n\t\t\t\taddr := cs.ServiceAddress\n\t\t\t\tif addr == \"\" {\n\t\t\t\t\taddr = cs.Address\n\t\t\t\t}\n\t\t\t\tif localNode.Addr.String() == addr && int(localNode.Port) == cs.ServicePort {\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\t\t\t\tserverAddr := net.JoinHostPort(addr, port)\n\t\t\t\tnomadServerServices = append(nomadServerServices, serverAddr)\n\t\t\t}\n\t\t}","sourceCodeStart":1053,"sourceCodeEnd":1089,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/server.go#L1053-L1089","documentation":"Also inside setupBootstrapHandler's fallback path: after listing datacenters, the server queries Consul for the \"nomad\" service (serf tag) in each DC via s.consulCatalog.Service. Failures are accumulated per datacenter into a multierror and returned, then the poll retries later. One bad DC does not stop the loop; the aggregate error reports all failing DCs.","triggerScenarios":"Quorum lost (fallback active), and consulCatalog.Service() errors for a DC — Consul API down, ACL token missing service read, query timeout against remote DCs (WAN issues), TLS problems.","commonSituations":"Multi-DC deployments where a remote DC's Consul is unreachable over the WAN, stale ACL tokens after rotation, Consul servers in one region down.","solutions":["Fix Consul connectivity/health in the DC(s) named in the error.","Verify the Consul ACL token can read the nomad service (service:read).","Check WAN gossip/connectivity between datacenters and Consul query timeouts.","Restore Nomad quorum so the Consul bootstrap fallback stops being used."],"exampleFix":"// before\n# ACL policy missing service read\nnode \"\" { policy = \"read\" }\n// after\nservice \"nomad\" { policy = \"read\" }\nnode \"\" { policy = \"read\" }","handlingStrategy":"retry","validationCode":"// preflight per DC\nfor _, dc := range dcs {\n    _, _, err := consulClient.Catalog().Service(\"nomad\", consul.ServiceTagSerf, &api.QueryOptions{Datacenter: dc})\n    if err != nil { log.Printf(\"dc %s not queryable: %v\", dc, err) }\n}","typeGuard":null,"tryCatchPattern":"// handler retries automatically; in automation treat this as transient\nif strings.Contains(errLog, \"failed to query service\") {\n    waitForConsulHealth(dcName, 5*time.Minute)\n}","preventionTips":["Keep ACL tokens valid across DCs (replication enabled).","Alert on WAN gossip degradation between Consul DCs.","Purge dead nomad service registrations regularly."],"tags":["consul","service-discovery","multi-dc","acl"],"backgroundTag":"consul-service-query-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}