hashicorp/nomad · error

failed to query Consul checks: %w

Error message

failed to query Consul checks: %w

What it means

During full sync, ChecksWithFilterOpts is called per namespace to enumerate all Consul checks. On failure the error is wrapped as 'failed to query Consul checks' and either returned directly or accumulated into a multierror depending on prior deregistration failures (mErr).

Source

Thrown at command/agent/consul/service_client.go:1100

				}); err != nil {
				metrics.IncrCounter([]string{"client", "consul", "sync_failure"}, 1)
				mErr = multierror.Append(mErr, err)
				fails++
				continue
			}
			sreg++
			metrics.IncrCounter([]string{"client", "consul", "service_registrations"}, 1)
		}

	}

	// Note: this query has to use the Nomad agent's own Consul token
	checksInConsul := make(map[string]*api.AgentCheck)
	for _, namespace := range namespaces {
		nsChecks, err := c.agentAPI.ChecksWithFilterOpts("", &api.QueryOptions{Namespace: normalizeNamespace(namespace)})
		if err != nil {
			metrics.IncrCounter([]string{"client", "consul", "sync_failure"}, 1)
			err = fmt.Errorf("failed to query Consul checks: %w", err)
			if mErr == nil || mErr.Len() == 0 {
				return err
			} else {
				mErr = multierror.Append(mErr, err)
				return mErr.ErrorOrNil()
			}
		}
		maps.Copy(checksInConsul, nsChecks)
	}

	// Remove Nomad checks in Consul but unknown locally
	for id, check := range checksInConsul {
		if _, ok := c.checks[id]; ok {
			// Known check, leave it
			continue
		}

		// Ignore if this is not a Nomad managed check. Also ignore

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Grant the Nomad token check:read (and agent read) permissions across namespaces.
2. Verify Consul agent health and connectivity.
3. Inspect the multierror output for concurrent deregistration failures and resolve the underlying Consul ACL/connectivity cause.

if _, err := agent.ChecksWithFilterOpts("", &api.QueryOptions{Namespace: ns}); err != nil {
  return fmt.Errorf("cannot read checks in ns %s: %w", ns, err)
}

nsChecks, err := c.agentAPI.ChecksWithFilterOpts("", qopts)
if err != nil {
  wrapped := fmt.Errorf("failed to query Consul checks: %w", err)
  if mErr == nil || mErr.Len() == 0 {
    return wrapped
  }
  mErr = multierror.Append(mErr, wrapped)
  return mErr.ErrorOrNil()
}

When it happens

Trigger: c.agentAPI.ChecksWithFilterOpts("", opts) errors for any namespace — ACL lacking check/session read rights, unreachable Consul agent, or transient Consul API errors during deregistration processing.

Common situations: Nomad Consul token missing the checks read permission; Consul agent down mid-sync; mixed multierror paths when both deregistrations and check queries fail simultaneously.

Related errors

• failed to query Consul namespaces: %w
• failed to query Consul services: %w
• failed to retrieve namespaces from consul: %w
• failed to retrieve services from consul: %w
• no one-time token returned

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/939dfdcfdffe69ab. Report an issue: GitHub.