hashicorp/nomad · error

failed to retrieve checks from consul: %w

Error message

failed to retrieve checks from consul: %w

What it means

Nomad's Consul client wrapper (checkQuery via service_client.go) fails to fetch health checks from the Consul agent API using ChecksWithFilterOpts. This wraps the underlying Consul API error so callers see which stage of service discovery failed. It is always a proxy for a real Consul connectivity or API error.

Source

Thrown at command/agent/consul/service_client.go:1799

	services := make(map[string]*api.AgentService)
	checks := make(map[string]*api.AgentCheck)

	// Query the services and checks to populate the allocation registrations.
	// Note: these queries have to use the Nomad agent's own Consul token
	for _, namespace := range namespaces {
		qo := &api.QueryOptions{
			Namespace: normalizeNamespace(namespace),
		}

		nsServices, err := c.agentAPI.ServicesWithFilterOpts("", qo)
		if err != nil {
			return nil, fmt.Errorf("failed to retrieve services from consul: %w", err)
		}
		maps.Copy(services, nsServices)

		nsChecks, err := c.agentAPI.ChecksWithFilterOpts("", qo)
		if err != nil {
			return nil, fmt.Errorf("failed to retrieve checks from consul: %w", err)
		}
		maps.Copy(checks, nsChecks)
	}

	// Populate the object
	for _, treg := range reg.Tasks {
		for serviceID, sreg := range treg.Services {
			sreg.Service = services[serviceID]
			for checkID := range sreg.CheckIDs {
				if check, ok := checks[checkID]; ok {
					sreg.Checks = append(sreg.Checks, check)
				}
			}

			if sidecarService := getNomadSidecar(serviceID, services); sidecarService != nil {
				sreg.SidecarService = sidecarService
				for _, check := range checks {
					if check.ServiceID == sidecarService.ID {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify the Consul agent is running and reachable at the configured consul.address (curl http://127.0.0.1:8500/v1/agent/services).
2. Check the wrapped error for 403 ACL errors and ensure the Nomad Consul token has service:read on the relevant namespaces.
3. Verify consul.tls / ca_file / cert settings match the Consul agent's TLS configuration.
4. Retry after restoring Consul connectivity; Nomad will reconcile on the next sync tick.

Example fix

// before
consul { address = "unix:///var/run/consul.sock" }
// after
consul { address = "127.0.0.1:8500", token = "<acl-token-with-service-read>" }

// shell
curl -sf http://${CONSUL_HTTP_ADDR:-127.0.0.1:8500}/v1/agent/services -H "X-Consul-Token: $CONSUL_HTTP_TOKEN" || echo 'consul agent unreachable'

try {
  checks := getConsulChecks()
} catch err {
  if strings.Contains(err.Error(), "failed to retrieve checks from consul") {
    log.Warn("consul unavailable; will retry on next sync", "cause", err)
    time.Sleep(backoff)
  }
}

When it happens

Trigger: Calling agent service discovery paths (e.g. task/group service registration reconciliation or `nomad agent` handling alloc services) when c.agentAPI.ChecksWithFilterOpts returns an error — Consul agent unreachable, HTTP error from Consul, or bad query options.

Common situations: Consul agent is down or restarted, wrong consul.address configured, ACL token lacks service:read, TLS misconfiguration between Nomad and Consul, network partition in multi-DC setups.

Related errors

• client.consul: unable to query Consul datacenters: %v
• error renewing the lease: %w
• failed to derive Consul token for task %s: %v
• failed to derive Consul token for service %s: %v
• unable to create unix socket for Consul HTTP endpoint: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/c8af328f2e546b74. Report an issue: GitHub.