hashicorp/nomad · error

unable to find configuration for consul cluster %v

Error message

unable to find configuration for consul cluster %v

What it means

clientForCluster looks up the requested Consul cluster in the hook's consulConfigs map; if absent it returns 'unable to find configuration for consul cluster %v'. This is the canonical internal lookup failure behind token derivation, and is also reachable from revokeTokens when cleaning up tokens for a cluster that no longer exists in config.

Source

Thrown at client/allocrunner/consul_hook.go:272

func (h *consulHook) getConsulToken(cluster string, req consul.JWTLoginRequest) (*consulapi.ACLToken, error) {
	client, err := h.clientForCluster(cluster)
	if err != nil {
		return nil, fmt.Errorf("failed to retrieve Consul client for cluster %s: %v", cluster, err)
	}

	t, err := client.DeriveTokenWithJWT(req)
	if err == nil {
		err = client.TokenPreflightCheck(h.shutdownCtx, t)
	}

	return t, err
}

func (h *consulHook) clientForCluster(cluster string) (consul.Client, error) {
	consulConf, ok := h.consulConfigs[cluster]
	if !ok {
		return nil, fmt.Errorf("unable to find configuration for consul cluster %v", cluster)
	}

	return h.consulClientConstructor(consulConf, h.logger)
}

// Postrun cleans up the Consul tokens after the tasks have exited.
func (h *consulHook) Postrun() error {
	return h.Destroy()
}

// Shutdown will get called when the client is gracefully stopping.
func (h *consulHook) Shutdown() {
	h.shutdownCancelFn()
}

// Destroy cleans up any remaining Consul tokens if the alloc is GC'd or fails
// to restore after a client restart.
func (h *consulHook) Destroy() error {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Compare the cluster name in the message with the client agent's configured consul cluster names
2. Add the missing cluster to the client configuration and restart nomad before retrying allocations
3. If raised during Postrun/revokeTokens, accept token cleanup is best-effort: tokens will expire by TTL; fix config for future runs
4. Ensure all clients a job can be scheduled onto define every consul.cluster the job references

Example fix

// before: client agent missing the named cluster
consul { name = "default" }
// after: define every cluster the jobs reference
consul { name = "default" }
consul { name = "prod-east" }

func clusterConfigured(consulConfigs map[string]consul.Config, cluster string) bool {
    _, ok := consulConfigs[cluster]
    return ok
}
// call before any token operation:
if !clusterConfigured(h.consulConfigs, cluster) {
    return fmt.Errorf("cluster %q not configured", cluster)
}

func clusterConfigured(configs map[string]consul.Config, cluster string) bool {
    _, ok := configs[cluster]
    return ok
}

if err := hook.Postrun(); err != nil {
    if strings.Contains(err.Error(), "unable to find configuration for consul cluster") {
        // best-effort revoke failed: log and continue; tokens expire via TTL
    }
}

When it happens

Trigger: getConsulToken called with a cluster name not in h.consulConfigs; revokeTokens (Postrun) attempts to revoke tokens for a cluster whose configuration was removed from the client between Prerun and Postrun.

Common situations: Client agent consul config changed/restarted between alloc start and stop; job references a cluster never configured on the client; multi-cluster naming mismatch after upgrading Nomad.

Related errors

• failed to retrieve Consul client for cluster %s: %v
• error creating bootstrap configuration for Connect proxy sid
• client stopped and may not longer create config entries
• non-default Consul cluster requires Nomad Enterprise
• expose may only be set for Consul service checks

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/e6d7aec3bc51b2f8. Report an issue: GitHub.