{"record":{"id":"a3ff4b46a77c2f50","repo":"hashicorp/nomad","slug":"failed-to-query-consul-services-w","errorCode":null,"errorMessage":"failed to query Consul services: %w","messagePattern":"failed to query Consul services: %w","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"command/agent/consul/service_client.go","lineNumber":1017,"sourceCode":"func (c *ServiceClient) sync(reason syncReason) error {\n\tc.logger.Trace(\"execute sync\", \"reason\", reason)\n\n\tsreg, creg, sdereg, cdereg, fails := 0, 0, 0, 0, 0\n\n\t// Get the list of all namespaces created so we can iterate them.\n\tnamespaces, err := c.namespacesClient.List()\n\tif err != nil {\n\t\tmetrics.IncrCounter([]string{\"client\", \"consul\", \"sync_failure\"}, 1)\n\t\treturn fmt.Errorf(\"failed to query Consul namespaces: %w\", err)\n\t}\n\n\t// Accumulate all services in Consul across all namespaces.\n\t// Note: this query has to use the Nomad agent's own Consul token\n\tservicesInConsul := make(map[string]*api.AgentService)\n\tfor _, namespace := range namespaces {\n\t\tif nsServices, err := c.agentAPI.ServicesWithFilterOpts(\"\", &api.QueryOptions{Namespace: normalizeNamespace(namespace)}); err != nil {\n\t\t\tmetrics.IncrCounter([]string{\"client\", \"consul\", \"sync_failure\"}, 1)\n\t\t\treturn fmt.Errorf(\"failed to query Consul services: %w\", err)\n\t\t} else {\n\t\t\tmaps.Copy(servicesInConsul, nsServices)\n\t\t}\n\t}\n\n\t// Compute whether we are still in probation period where we will avoid\n\t// de-registering services.\n\tinProbation := time.Now().Before(c.deregisterProbationExpiry)\n\n\tvar mErr *multierror.Error // collect errors for individual services/checks\n\n\t// Remove Nomad services in Consul but unknown to Nomad.\n\tfor id, service := range servicesInConsul {\n\t\tif _, ok := c.services[id]; ok {\n\t\t\t// Known service, skip\n\t\t\tcontinue\n\t\t}\n","sourceCodeStart":999,"sourceCodeEnd":1035,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/command/agent/consul/service_client.go#L999-L1035","documentation":"While accumulating all services across Consul namespaces during a full sync, ServicesWithFilterOpts is called per namespace using the agent's own Consul token. Any per-namespace query failure aborts the sync with this wrapped error.","triggerScenarios":"c.agentAPI.ServicesWithFilterOpts(\"\", opts) fails for a namespace — ACL token lacks service:read in that namespace, Consul agent unreachable, or the namespace was deleted between listing and querying.","commonSituations":"Nomad's Consul token missing service:read in non-default namespaces; Consul agent connectivity issues; namespace churn during sync (deleted namespace still in list).","solutions":["Update the Nomad Consul ACL token to include service:read across all relevant namespaces.","Check Consul agent availability and network paths.","Re-run the sync after stale namespaces are removed; verify namespace list is current."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"for _, ns := range namespaces {\n  if _, err := agent.ServicesWithFilterOpts(\"\", &api.QueryOptions{Namespace: ns}); err != nil {\n    return fmt.Errorf(\"token lacks service read in ns %s: %w\", ns, err)\n  }\n}","typeGuard":null,"tryCatchPattern":"nsServices, err := c.agentAPI.ServicesWithFilterOpts(\"\", qopts)\nif err != nil {\n  return backoff.Retry(func() error {\n    _, err := c.agentAPI.ServicesWithFilterOpts(\"\", qopts)\n    return err\n  }, retryPolicy)\n}","preventionTips":["Grant service:read across all namespaces to the Nomad token.","Watch for namespace deletion during syncs.","Ensure Consul agent liveness checks before sync windows."],"tags":["consul","services","acl","sync","nomad"],"backgroundTag":"consul-api-permission-denied","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}