{"record":{"id":"07e9bcea4ad0c7c3","repo":"hashicorp/nomad","slug":"failed-to-retrieve-consul-client-for-cluster-s","errorCode":null,"errorMessage":"failed to retrieve Consul client for cluster %s: %v","messagePattern":"failed to retrieve Consul client for cluster (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/allocrunner/consul_hook.go","lineNumber":258,"sourceCode":"\t\t\t}\n\n\t\t}\n\n\t\t// Store token in results.\n\t\tif _, ok = tokens[clusterName]; !ok {\n\t\t\ttokens[clusterName] = make(map[string]*consulapi.ACLToken)\n\t\t}\n\n\t\ttokens[clusterName][tokenName] = token\n\t}\n\n\treturn mErr.ErrorOrNil()\n}\n\nfunc (h *consulHook) getConsulToken(cluster string, req consul.JWTLoginRequest) (*consulapi.ACLToken, error) {\n\tclient, err := h.clientForCluster(cluster)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to retrieve Consul client for cluster %s: %v\", cluster, err)\n\t}\n\n\tt, err := client.DeriveTokenWithJWT(req)\n\tif err == nil {\n\t\terr = client.TokenPreflightCheck(h.shutdownCtx, t)\n\t}\n\n\treturn t, err\n}\n\nfunc (h *consulHook) clientForCluster(cluster string) (consul.Client, error) {\n\tconsulConf, ok := h.consulConfigs[cluster]\n\tif !ok {\n\t\treturn nil, fmt.Errorf(\"unable to find configuration for consul cluster %v\", cluster)\n\t}\n\n\treturn h.consulClientConstructor(consulConf, h.logger)\n}","sourceCodeStart":240,"sourceCodeEnd":276,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/allocrunner/consul_hook.go#L240-L276","documentation":"getConsulToken first resolves a Consul client for the named cluster via clientForCluster. If that lookup fails, the error is wrapped as 'failed to retrieve Consul client for cluster %s'. The wrapped error is almost always 'unable to find configuration for consul cluster', i.e. the client-side Consul config map lacks that cluster.","triggerScenarios":"Called from prepareConsulTokensForTask or prepareConsulTokensForServices after a JWTLoginRequest is built, when the cluster name passed in has no entry in h.consulConfigs (or the consulClientConstructor itself fails).","commonSituations":"Same config drift as the 'no such consul cluster' errors but surfaced one layer deeper; cluster removed from client config while allocs still reference it; constructor failure due to invalid Consul client options (bad address/TLS settings).","solutions":["Check the wrapped error: if it is the config lookup, align the job's consul.cluster with the client agent's configured clusters","Validate the Consul client options (address, TLS) if the constructor itself failed","Update client agent config to include the missing cluster and restart nomad","Reschedule affected allocations after the config fix"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// before deriving tokens, ensure the cluster resolves to a configured client\nconsulConf, ok := h.consulConfigs[cluster]\nif !ok {\n    return fmt.Errorf(\"cannot derive token: consul cluster %q unconfigured\", cluster)\n}","typeGuard":null,"tryCatchPattern":"if err := hook.Prerun(); err != nil {\n    if strings.Contains(err.Error(), \"failed to retrieve Consul client for cluster\") {\n        // unwrap to see if it's the config lookup or constructor; fix config for lookup,\n        // fix consul address/TLS options for constructor failures\n    }\n}","preventionTips":["Ensure every consul.cluster referenced by jobs exists in client agent config","Validate consul client options (address, TLS certs) in agent config before rollout","Re-run Prerun only after client config changes are complete","Test token derivation on a canary client before fleet-wide config changes"],"tags":["consul","client-configuration","nomad","token-derivation"],"backgroundTag":"missing-cluster-configuration","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}