{"record":{"id":"939dfdcfdffe69ab","repo":"hashicorp/nomad","slug":"failed-to-query-consul-checks-w","errorCode":null,"errorMessage":"failed to query Consul checks: %w","messagePattern":"failed to query Consul checks: %w","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"command/agent/consul/service_client.go","lineNumber":1100,"sourceCode":"\t\t\t\t}); err != nil {\n\t\t\t\tmetrics.IncrCounter([]string{\"client\", \"consul\", \"sync_failure\"}, 1)\n\t\t\t\tmErr = multierror.Append(mErr, err)\n\t\t\t\tfails++\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\tsreg++\n\t\t\tmetrics.IncrCounter([]string{\"client\", \"consul\", \"service_registrations\"}, 1)\n\t\t}\n\n\t}\n\n\t// Note: this query has to use the Nomad agent's own Consul token\n\tchecksInConsul := make(map[string]*api.AgentCheck)\n\tfor _, namespace := range namespaces {\n\t\tnsChecks, err := c.agentAPI.ChecksWithFilterOpts(\"\", &api.QueryOptions{Namespace: normalizeNamespace(namespace)})\n\t\tif err != nil {\n\t\t\tmetrics.IncrCounter([]string{\"client\", \"consul\", \"sync_failure\"}, 1)\n\t\t\terr = fmt.Errorf(\"failed to query Consul checks: %w\", err)\n\t\t\tif mErr == nil || mErr.Len() == 0 {\n\t\t\t\treturn err\n\t\t\t} else {\n\t\t\t\tmErr = multierror.Append(mErr, err)\n\t\t\t\treturn mErr.ErrorOrNil()\n\t\t\t}\n\t\t}\n\t\tmaps.Copy(checksInConsul, nsChecks)\n\t}\n\n\t// Remove Nomad checks in Consul but unknown locally\n\tfor id, check := range checksInConsul {\n\t\tif _, ok := c.checks[id]; ok {\n\t\t\t// Known check, leave it\n\t\t\tcontinue\n\t\t}\n\n\t\t// Ignore if this is not a Nomad managed check. Also ignore","sourceCodeStart":1082,"sourceCodeEnd":1118,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/command/agent/consul/service_client.go#L1082-L1118","documentation":"During full sync, ChecksWithFilterOpts is called per namespace to enumerate all Consul checks. On failure the error is wrapped as 'failed to query Consul checks' and either returned directly or accumulated into a multierror depending on prior deregistration failures (mErr).","triggerScenarios":"c.agentAPI.ChecksWithFilterOpts(\"\", opts) errors for any namespace — ACL lacking check/session read rights, unreachable Consul agent, or transient Consul API errors during deregistration processing.","commonSituations":"Nomad Consul token missing the checks read permission; Consul agent down mid-sync; mixed multierror paths when both deregistrations and check queries fail simultaneously.","solutions":["Grant the Nomad token check:read (and agent read) permissions across namespaces.","Verify Consul agent health and connectivity.","Inspect the multierror output for concurrent deregistration failures and resolve the underlying Consul ACL/connectivity cause."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"if _, err := agent.ChecksWithFilterOpts(\"\", &api.QueryOptions{Namespace: ns}); err != nil {\n  return fmt.Errorf(\"cannot read checks in ns %s: %w\", ns, err)\n}","typeGuard":null,"tryCatchPattern":"nsChecks, err := c.agentAPI.ChecksWithFilterOpts(\"\", qopts)\nif err != nil {\n  wrapped := fmt.Errorf(\"failed to query Consul checks: %w\", err)\n  if mErr == nil || mErr.Len() == 0 {\n    return wrapped\n  }\n  mErr = multierror.Append(mErr, wrapped)\n  return mErr.ErrorOrNil()\n}","preventionTips":["Include check:read in the Nomad Consul ACL policy.","Track sync_failure metrics for early detection.","Handle multierror returns rather than only single errors."],"tags":["consul","checks","acl","sync","multierror","nomad"],"backgroundTag":"consul-api-permission-denied","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}