hashicorp/nomad · error

failed to retrieve Consul client for cluster %s: %v

Error message

failed to retrieve Consul client for cluster %s: %v

What it means

getConsulToken first resolves a Consul client for the named cluster via clientForCluster. If that lookup fails, the error is wrapped as 'failed to retrieve Consul client for cluster %s'. The wrapped error is almost always 'unable to find configuration for consul cluster', i.e. the client-side Consul config map lacks that cluster.

Source

Thrown at client/allocrunner/consul_hook.go:258

			}

		}

		// Store token in results.
		if _, ok = tokens[clusterName]; !ok {
			tokens[clusterName] = make(map[string]*consulapi.ACLToken)
		}

		tokens[clusterName][tokenName] = token
	}

	return mErr.ErrorOrNil()
}

func (h *consulHook) getConsulToken(cluster string, req consul.JWTLoginRequest) (*consulapi.ACLToken, error) {
	client, err := h.clientForCluster(cluster)
	if err != nil {
		return nil, fmt.Errorf("failed to retrieve Consul client for cluster %s: %v", cluster, err)
	}

	t, err := client.DeriveTokenWithJWT(req)
	if err == nil {
		err = client.TokenPreflightCheck(h.shutdownCtx, t)
	}

	return t, err
}

func (h *consulHook) clientForCluster(cluster string) (consul.Client, error) {
	consulConf, ok := h.consulConfigs[cluster]
	if !ok {
		return nil, fmt.Errorf("unable to find configuration for consul cluster %v", cluster)
	}

	return h.consulClientConstructor(consulConf, h.logger)
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the wrapped error: if it is the config lookup, align the job's consul.cluster with the client agent's configured clusters
2. Validate the Consul client options (address, TLS) if the constructor itself failed
3. Update client agent config to include the missing cluster and restart nomad
4. Reschedule affected allocations after the config fix

// before deriving tokens, ensure the cluster resolves to a configured client
consulConf, ok := h.consulConfigs[cluster]
if !ok {
    return fmt.Errorf("cannot derive token: consul cluster %q unconfigured", cluster)
}

if err := hook.Prerun(); err != nil {
    if strings.Contains(err.Error(), "failed to retrieve Consul client for cluster") {
        // unwrap to see if it's the config lookup or constructor; fix config for lookup,
        // fix consul address/TLS options for constructor failures
    }
}

When it happens

Trigger: Called from prepareConsulTokensForTask or prepareConsulTokensForServices after a JWTLoginRequest is built, when the cluster name passed in has no entry in h.consulConfigs (or the consulClientConstructor itself fails).

Common situations: Same config drift as the 'no such consul cluster' errors but surfaced one layer deeper; cluster removed from client config while allocs still reference it; constructor failure due to invalid Consul client options (bad address/TLS settings).

Related errors

• unable to find configuration for consul cluster %v
• error creating bootstrap configuration for Connect proxy sid
• client stopped and may not longer create config entries
• non-default Consul cluster requires Nomad Enterprise
• expose may only be set for Consul service checks

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/07e9bcea4ad0c7c3. Report an issue: GitHub.