{"record":{"id":"c8af328f2e546b74","repo":"hashicorp/nomad","slug":"failed-to-retrieve-checks-from-consul-w","errorCode":null,"errorMessage":"failed to retrieve checks from consul: %w","messagePattern":"failed to retrieve checks from consul: %w","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"command/agent/consul/service_client.go","lineNumber":1799,"sourceCode":"\tservices := make(map[string]*api.AgentService)\n\tchecks := make(map[string]*api.AgentCheck)\n\n\t// Query the services and checks to populate the allocation registrations.\n\t// Note: these queries have to use the Nomad agent's own Consul token\n\tfor _, namespace := range namespaces {\n\t\tqo := &api.QueryOptions{\n\t\t\tNamespace: normalizeNamespace(namespace),\n\t\t}\n\n\t\tnsServices, err := c.agentAPI.ServicesWithFilterOpts(\"\", qo)\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"failed to retrieve services from consul: %w\", err)\n\t\t}\n\t\tmaps.Copy(services, nsServices)\n\n\t\tnsChecks, err := c.agentAPI.ChecksWithFilterOpts(\"\", qo)\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"failed to retrieve checks from consul: %w\", err)\n\t\t}\n\t\tmaps.Copy(checks, nsChecks)\n\t}\n\n\t// Populate the object\n\tfor _, treg := range reg.Tasks {\n\t\tfor serviceID, sreg := range treg.Services {\n\t\t\tsreg.Service = services[serviceID]\n\t\t\tfor checkID := range sreg.CheckIDs {\n\t\t\t\tif check, ok := checks[checkID]; ok {\n\t\t\t\t\tsreg.Checks = append(sreg.Checks, check)\n\t\t\t\t}\n\t\t\t}\n\n\t\t\tif sidecarService := getNomadSidecar(serviceID, services); sidecarService != nil {\n\t\t\t\tsreg.SidecarService = sidecarService\n\t\t\t\tfor _, check := range checks {\n\t\t\t\t\tif check.ServiceID == sidecarService.ID {","sourceCodeStart":1781,"sourceCodeEnd":1817,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/command/agent/consul/service_client.go#L1781-L1817","documentation":"Nomad's Consul client wrapper (checkQuery via service_client.go) fails to fetch health checks from the Consul agent API using ChecksWithFilterOpts. This wraps the underlying Consul API error so callers see which stage of service discovery failed. It is always a proxy for a real Consul connectivity or API error.","triggerScenarios":"Calling agent service discovery paths (e.g. task/group service registration reconciliation or `nomad agent` handling alloc services) when c.agentAPI.ChecksWithFilterOpts returns an error — Consul agent unreachable, HTTP error from Consul, or bad query options.","commonSituations":"Consul agent is down or restarted, wrong consul.address configured, ACL token lacks service:read, TLS misconfiguration between Nomad and Consul, network partition in multi-DC setups.","solutions":["Verify the Consul agent is running and reachable at the configured consul.address (curl http://127.0.0.1:8500/v1/agent/services).","Check the wrapped error for 403 ACL errors and ensure the Nomad Consul token has service:read on the relevant namespaces.","Verify consul.tls / ca_file / cert settings match the Consul agent's TLS configuration.","Retry after restoring Consul connectivity; Nomad will reconcile on the next sync tick."],"exampleFix":"// before\nconsul { address = \"unix:///var/run/consul.sock\" }\n// after\nconsul { address = \"127.0.0.1:8500\", token = \"<acl-token-with-service-read>\" }","handlingStrategy":"retry","validationCode":"// shell\ncurl -sf http://${CONSUL_HTTP_ADDR:-127.0.0.1:8500}/v1/agent/services -H \"X-Consul-Token: $CONSUL_HTTP_TOKEN\" || echo 'consul agent unreachable'","typeGuard":null,"tryCatchPattern":"try {\n  checks := getConsulChecks()\n} catch err {\n  if strings.Contains(err.Error(), \"failed to retrieve checks from consul\") {\n    log.Warn(\"consul unavailable; will retry on next sync\", \"cause\", err)\n    time.Sleep(backoff)\n  }\n}","preventionTips":["Monitor Consul agent health from every Nomad node (consul members / health endpoint).","Ensure the Nomad Consul ACL token has service:read on all namespaces used.","Keep consul.tls and address settings identical across Nomad configs.","Test connectivity with curl before rolling Nomad upgrades."],"tags":["consul","service-discovery","network"],"backgroundTag":"consul-api-unreachable","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}