hashicorp/nomad · error

failed to retrieve services from consul: %w

Error message

failed to retrieve services from consul: %w

What it means

While enumerating every Consul namespace to gather all agent services, Nomad calls agentAPI.ServicesWithFilterOpts per namespace. Any failure from the Consul agent API (ACL denial on service:read, agent unreachable, malformed response) is wrapped in this message and AllocRegistrations returns nil, preventing reconciliation of service registrations.

Source

Thrown at command/agent/consul/service_client.go:1793

	// Get the list of all namespaces created so we can iterate them.
	namespaces, err := c.namespacesClient.List()
	if err != nil {
		return nil, fmt.Errorf("failed to retrieve namespaces from consul: %w", err)
	}

	services := make(map[string]*api.AgentService)
	checks := make(map[string]*api.AgentCheck)

	// Query the services and checks to populate the allocation registrations.
	// Note: these queries have to use the Nomad agent's own Consul token
	for _, namespace := range namespaces {
		qo := &api.QueryOptions{
			Namespace: normalizeNamespace(namespace),
		}

		nsServices, err := c.agentAPI.ServicesWithFilterOpts("", qo)
		if err != nil {
			return nil, fmt.Errorf("failed to retrieve services from consul: %w", err)
		}
		maps.Copy(services, nsServices)

		nsChecks, err := c.agentAPI.ChecksWithFilterOpts("", qo)
		if err != nil {
			return nil, fmt.Errorf("failed to retrieve checks from consul: %w", err)
		}
		maps.Copy(checks, nsChecks)
	}

	// Populate the object
	for _, treg := range reg.Tasks {
		for serviceID, sreg := range treg.Services {
			sreg.Service = services[serviceID]
			for checkID := range sreg.CheckIDs {
				if check, ok := checks[checkID]; ok {
					sreg.Checks = append(sreg.Checks, check)
				}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Grant the Nomad Consul token service:read (service:write) in every namespace Nomad manages, or restrict consul namespaces config to ones it can read
2. Verify the Consul agent HTTP address/port in the Nomad consul stanza and test with `consul catalog services` using the same token
3. Retry if transient — reconciliation reruns periodically; a persistent error indicates ACL/address issues
4. Check the wrapped %w error string for 'Permission denied' (ACL) vs connection errors

Example fix

// before (Consul ACL policy)
namespace "nomad" {
  // no service permissions
}
// after
namespace "nomad" {
  service_prefix "" {
    policy = "write"
  }
  key_prefix "" {
    policy = "read"
  }
}

// Pre-flight the same API call the client makes, using the Nomad token
_, _, err := consulClient.Agent().ServicesWithFilterOpts("", &api.QueryOptions{Namespace: ns})
if err != nil {
  log.Printf("token cannot read services in ns %q: %v", ns, err)
}

regs, err := client.AllocRegistrations(allocID)
if err != nil {
  if strings.Contains(err.Error(), "failed to retrieve services") {
    // transient or ACL: schedule a retry / alert on repeated failure
  }
}

When it happens

Trigger: AllocRegistrations iterates namespaces; for each, c.agentAPI.ServicesWithFilterOpts("", qo) with a namespace-scoped QueryOptions errors — typically ACL token missing service:read in that namespace or Consul agent connection failure.

Common situations: Wildcard/multi-namespace setups where the Nomad token lacks service:read in one namespace; Consul agent restarted or listening on a different address; transient network blip during reconciliation causing spurious errors.

Related errors

• client.consul: unable to query Consul datacenters: %v
• unable to query service %+q from Consul datacenter %+q: %v
• no Nomad Servers advertising service %q in Consul datacenter
• failed to query Consul namespaces: %w
• failed to query Consul services: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/58ec4f3dd203574d. Report an issue: GitHub.