{"record":{"id":"1d16f65562362c3b","repo":"hashicorp/nomad","slug":"failed-to-query-consul-namespaces-w","errorCode":null,"errorMessage":"failed to query Consul namespaces: %w","messagePattern":"failed to query Consul namespaces: %w","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"command/agent/consul/service_client.go","lineNumber":1008,"sourceCode":"\tfor _, cid := range ops.deregChecks {\n\t\tdelete(c.checks, cid)\n\t\tc.explicitlyDeregisteredChecks.Insert(cid)\n\t}\n\tmetrics.SetGauge([]string{\"client\", \"consul\", \"services\"}, float32(len(c.services)))\n\tmetrics.SetGauge([]string{\"client\", \"consul\", \"checks\"}, float32(len(c.checks)))\n}\n\n// sync enqueued operations.\nfunc (c *ServiceClient) sync(reason syncReason) error {\n\tc.logger.Trace(\"execute sync\", \"reason\", reason)\n\n\tsreg, creg, sdereg, cdereg, fails := 0, 0, 0, 0, 0\n\n\t// Get the list of all namespaces created so we can iterate them.\n\tnamespaces, err := c.namespacesClient.List()\n\tif err != nil {\n\t\tmetrics.IncrCounter([]string{\"client\", \"consul\", \"sync_failure\"}, 1)\n\t\treturn fmt.Errorf(\"failed to query Consul namespaces: %w\", err)\n\t}\n\n\t// Accumulate all services in Consul across all namespaces.\n\t// Note: this query has to use the Nomad agent's own Consul token\n\tservicesInConsul := make(map[string]*api.AgentService)\n\tfor _, namespace := range namespaces {\n\t\tif nsServices, err := c.agentAPI.ServicesWithFilterOpts(\"\", &api.QueryOptions{Namespace: normalizeNamespace(namespace)}); err != nil {\n\t\t\tmetrics.IncrCounter([]string{\"client\", \"consul\", \"sync_failure\"}, 1)\n\t\t\treturn fmt.Errorf(\"failed to query Consul services: %w\", err)\n\t\t} else {\n\t\t\tmaps.Copy(servicesInConsul, nsServices)\n\t\t}\n\t}\n\n\t// Compute whether we are still in probation period where we will avoid\n\t// de-registering services.\n\tinProbation := time.Now().Before(c.deregisterProbationExpiry)\n","sourceCodeStart":990,"sourceCodeEnd":1026,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/command/agent/consul/service_client.go#L990-L1026","documentation":"During full sync of Consul services/checks, the service client first lists all Consul namespaces via the namespaces client. If that API call fails (ACL denial, unreachable Consul, enterprise-only endpoint), the sync aborts and returns this wrapped error and increments the sync_failure metric.","triggerScenarios":"c.namespacesClient.List() returns an error during sync — Consul agent unreachable, the Nomad Consul token lacks namespace:read permission, or the Consul edition lacks the namespace listing API (OSS Consul pre-1.7).","commonSituations":"Nomad client pointed at Consul with overly restrictive ACL tokens; Consul agent restarted/down; using open-source Consul without Enterprise namespaces configured while namespace features are expected.","solutions":["Grant the Nomad Consul token namespace listing permissions (namespace:list / operator namespace read).","Verify Consul agent connectivity and health from the Nomad client node.","Confirm Consul version/edition supports namespaces (Enterprise 1.7+); disable namespace expectations if running OSS."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// pre-check token capabilities\nnamespaces, err := consulClient.Namespaces().List(nil)\nif err != nil {\n  return fmt.Errorf(\"nomad token cannot list Consul namespaces: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"namespaces, err := c.namespacesClient.List()\nif err != nil {\n  if isPermissionDenied(err) {\n    // alert operator to fix Consul ACL token\n  }\n  return retry.After(backoff, fmt.Errorf(\"failed to query Consul namespaces: %w\", err))\n}","preventionTips":["Provision the Nomad Consul token with namespace list permissions.","Monitor client.consul.sync_failure metrics.","Verify Consul version/edition supports namespaces before enabling namespace features."],"tags":["consul","namespaces","acl","sync","nomad"],"backgroundTag":"consul-api-permission-denied","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}