hashicorp/nomad · error

failed to query Consul services: %w

Error message

failed to query Consul services: %w

What it means

While accumulating all services across Consul namespaces during a full sync, ServicesWithFilterOpts is called per namespace using the agent's own Consul token. Any per-namespace query failure aborts the sync with this wrapped error.

Source

Thrown at command/agent/consul/service_client.go:1017

func (c *ServiceClient) sync(reason syncReason) error {
	c.logger.Trace("execute sync", "reason", reason)

	sreg, creg, sdereg, cdereg, fails := 0, 0, 0, 0, 0

	// Get the list of all namespaces created so we can iterate them.
	namespaces, err := c.namespacesClient.List()
	if err != nil {
		metrics.IncrCounter([]string{"client", "consul", "sync_failure"}, 1)
		return fmt.Errorf("failed to query Consul namespaces: %w", err)
	}

	// Accumulate all services in Consul across all namespaces.
	// Note: this query has to use the Nomad agent's own Consul token
	servicesInConsul := make(map[string]*api.AgentService)
	for _, namespace := range namespaces {
		if nsServices, err := c.agentAPI.ServicesWithFilterOpts("", &api.QueryOptions{Namespace: normalizeNamespace(namespace)}); err != nil {
			metrics.IncrCounter([]string{"client", "consul", "sync_failure"}, 1)
			return fmt.Errorf("failed to query Consul services: %w", err)
		} else {
			maps.Copy(servicesInConsul, nsServices)
		}
	}

	// Compute whether we are still in probation period where we will avoid
	// de-registering services.
	inProbation := time.Now().Before(c.deregisterProbationExpiry)

	var mErr *multierror.Error // collect errors for individual services/checks

	// Remove Nomad services in Consul but unknown to Nomad.
	for id, service := range servicesInConsul {
		if _, ok := c.services[id]; ok {
			// Known service, skip
			continue
		}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Update the Nomad Consul ACL token to include service:read across all relevant namespaces.
2. Check Consul agent availability and network paths.
3. Re-run the sync after stale namespaces are removed; verify namespace list is current.

for _, ns := range namespaces {
  if _, err := agent.ServicesWithFilterOpts("", &api.QueryOptions{Namespace: ns}); err != nil {
    return fmt.Errorf("token lacks service read in ns %s: %w", ns, err)
  }
}

nsServices, err := c.agentAPI.ServicesWithFilterOpts("", qopts)
if err != nil {
  return backoff.Retry(func() error {
    _, err := c.agentAPI.ServicesWithFilterOpts("", qopts)
    return err
  }, retryPolicy)
}

When it happens

Trigger: c.agentAPI.ServicesWithFilterOpts("", opts) fails for a namespace — ACL token lacks service:read in that namespace, Consul agent unreachable, or the namespace was deleted between listing and querying.

Common situations: Nomad's Consul token missing service:read in non-default namespaces; Consul agent connectivity issues; namespace churn during sync (deleted namespace still in list).

Related errors

• failed to query Consul namespaces: %w
• failed to query Consul checks: %w
• error getting signed identity for service %s: %v
• failed to derive Consul token for service %s: %v
• failed to retrieve namespaces from consul: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/a3ff4b46a77c2f50. Report an issue: GitHub.