hashicorp/nomad · error

failed to retrieve namespaces from consul: %w

Error message

failed to retrieve namespaces from consul: %w

What it means

When collecting all alloc registrations for self-healing (serviceClient.AllocRegistrations), Nomad lists Consul namespaces via the namespaces client. If the Consul API returns an error (Enterprise feature, permissions, connectivity), this wrapper aborts the listing, so allocation service/check state cannot be reconciled.

Source

Thrown at command/agent/consul/service_client.go:1778

// AllocRegistrations returns the registrations for the given allocation. If the
// allocation has no registrations, the response is a nil object.
func (c *ServiceClient) AllocRegistrations(allocID string) (*serviceregistration.AllocRegistration, error) {
	// Get the internal struct using the lock
	c.allocRegistrationsLock.RLock()
	regInternal, ok := c.allocRegistrations[allocID]
	if !ok {
		c.allocRegistrationsLock.RUnlock()
		return nil, nil
	}

	// Copy so we don't expose internal structs
	reg := regInternal.Copy()
	c.allocRegistrationsLock.RUnlock()

	// Get the list of all namespaces created so we can iterate them.
	namespaces, err := c.namespacesClient.List()
	if err != nil {
		return nil, fmt.Errorf("failed to retrieve namespaces from consul: %w", err)
	}

	services := make(map[string]*api.AgentService)
	checks := make(map[string]*api.AgentCheck)

	// Query the services and checks to populate the allocation registrations.
	// Note: these queries have to use the Nomad agent's own Consul token
	for _, namespace := range namespaces {
		qo := &api.QueryOptions{
			Namespace: normalizeNamespace(namespace),
		}

		nsServices, err := c.agentAPI.ServicesWithFilterOpts("", qo)
		if err != nil {
			return nil, fmt.Errorf("failed to retrieve services from consul: %w", err)
		}
		maps.Copy(services, nsServices)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Use Consul Enterprise (>=1.7) or remove consul.namespace config if running Consul OSS
2. Grant the Nomad Consul ACL token namespace list/read permissions (operator namespace list, service:write per namespace)
3. Verify connectivity from the Nomad client to the Consul agent HTTP endpoint
4. Check the wrapped %w error for 403 vs 404 to distinguish ACL vs missing-feature

Example fix

# before (consul stanza)
consul {
  namespace = "nomad" # OSS Consul has no namespaces
}
# after
consul {
  # namespace removed for Consul OSS
}

// Pre-flight: confirm the Consul deployment supports namespaces and the token can list them
ns, _, err := consulClient.Namespaces().List(nil)
if err != nil {
  log.Printf("namespaces unavailable (OSS Consul or ACL gap): %v — unset consul.namespace", err)
}

regs, err := client.AllocRegistrations(allocID)
if err != nil && strings.Contains(err.Error(), "failed to retrieve namespaces") {
  // reconcile later; check ACL token / Consul edition
} else if err != nil {
  log.Fatal(err)
}

When it happens

Trigger: AllocRegistrations calls c.namespacesClient.List() which hits the Consul /v1/namespaces endpoint; any API error — 403 ACL lacking 'namespace:list', Consul OSS lacking namespaces endpoint (404), or network failure — is wrapped here.

Common situations: Nomad configured with consul.namespace on Consul OSS (namespaces unsupported); Nomad ACL token missing the namespace:read/list permissions; Consul agent unreachable; enterprise Consul older than 1.7.

Related errors

• failed to query Consul namespaces: %w
• failed to query Consul services: %w
• failed to query Consul checks: %w
• failed to retrieve services from consul: %w
• no one-time token returned

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/29cefc587db9a99f. Report an issue: GitHub.