hashicorp/nomad · error

failed to query Consul namespaces: %w

Error message

failed to query Consul namespaces: %w

What it means

During full sync of Consul services/checks, the service client first lists all Consul namespaces via the namespaces client. If that API call fails (ACL denial, unreachable Consul, enterprise-only endpoint), the sync aborts and returns this wrapped error and increments the sync_failure metric.

Source

Thrown at command/agent/consul/service_client.go:1008

	for _, cid := range ops.deregChecks {
		delete(c.checks, cid)
		c.explicitlyDeregisteredChecks.Insert(cid)
	}
	metrics.SetGauge([]string{"client", "consul", "services"}, float32(len(c.services)))
	metrics.SetGauge([]string{"client", "consul", "checks"}, float32(len(c.checks)))
}

// sync enqueued operations.
func (c *ServiceClient) sync(reason syncReason) error {
	c.logger.Trace("execute sync", "reason", reason)

	sreg, creg, sdereg, cdereg, fails := 0, 0, 0, 0, 0

	// Get the list of all namespaces created so we can iterate them.
	namespaces, err := c.namespacesClient.List()
	if err != nil {
		metrics.IncrCounter([]string{"client", "consul", "sync_failure"}, 1)
		return fmt.Errorf("failed to query Consul namespaces: %w", err)
	}

	// Accumulate all services in Consul across all namespaces.
	// Note: this query has to use the Nomad agent's own Consul token
	servicesInConsul := make(map[string]*api.AgentService)
	for _, namespace := range namespaces {
		if nsServices, err := c.agentAPI.ServicesWithFilterOpts("", &api.QueryOptions{Namespace: normalizeNamespace(namespace)}); err != nil {
			metrics.IncrCounter([]string{"client", "consul", "sync_failure"}, 1)
			return fmt.Errorf("failed to query Consul services: %w", err)
		} else {
			maps.Copy(servicesInConsul, nsServices)
		}
	}

	// Compute whether we are still in probation period where we will avoid
	// de-registering services.
	inProbation := time.Now().Before(c.deregisterProbationExpiry)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Grant the Nomad Consul token namespace listing permissions (namespace:list / operator namespace read).
2. Verify Consul agent connectivity and health from the Nomad client node.
3. Confirm Consul version/edition supports namespaces (Enterprise 1.7+); disable namespace expectations if running OSS.

// pre-check token capabilities
namespaces, err := consulClient.Namespaces().List(nil)
if err != nil {
  return fmt.Errorf("nomad token cannot list Consul namespaces: %w", err)
}

namespaces, err := c.namespacesClient.List()
if err != nil {
  if isPermissionDenied(err) {
    // alert operator to fix Consul ACL token
  }
  return retry.After(backoff, fmt.Errorf("failed to query Consul namespaces: %w", err))
}

When it happens

Trigger: c.namespacesClient.List() returns an error during sync — Consul agent unreachable, the Nomad Consul token lacks namespace:read permission, or the Consul edition lacks the namespace listing API (OSS Consul pre-1.7).

Common situations: Nomad client pointed at Consul with overly restrictive ACL tokens; Consul agent restarted/down; using open-source Consul without Enterprise namespaces configured while namespace features are expected.

Related errors

• failed to query Consul services: %w
• failed to query Consul checks: %w
• failed to retrieve namespaces from consul: %w
• failed to retrieve services from consul: %w
• no one-time token returned

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/1d16f65562362c3b. Report an issue: GitHub.