hashicorp/nomad · error

failed to derive Consul token for task %s: %v

Error message

failed to derive Consul token for task %s: %v

What it means

After building a JWTLoginRequest from the signed identity, the hook calls getConsulToken, which performs JWT login against Consul (DeriveTokenWithJWT) plus a token preflight check. Any failure is wrapped as 'failed to derive Consul token for task %s'. The root cause is in the wrapped error: unreachable Consul, wrong auth method, invalid/expired JWT, or ACL check failure.

Source

Thrown at client/allocrunner/consul_hook.go:179

		ti := *task.IdentityHandle(wid)
		swi, err := h.widmgr.Get(ti)
		if err != nil {
			return fmt.Errorf("error getting signed identity for task %s: %v", task.Name, err)
		}

		h.logger.Debug("logging into consul", "name", ti.IdentityName, "type", ti.WorkloadType)
		req := consul.JWTLoginRequest{
			JWT:            swi.JWT,
			AuthMethodName: consulConfig.TaskIdentityAuthMethod,
			Meta: map[string]string{
				"requested_by": fmt.Sprintf("nomad_task_%s", task.Name),
				"node_id":      h.alloc.NodeID,
			},
		}

		token, err = h.getConsulToken(consulConfig.Name, req)
		if err != nil {
			return fmt.Errorf("failed to derive Consul token for task %s: %v", task.Name, err)
		}
	}

	// Store token in results.
	if _, ok = tokens[clusterName]; !ok {
		tokens[clusterName] = make(map[string]*consulapi.ACLToken)
	}

	tokens[clusterName][tokenName] = token

	return nil
}

func (h *consulHook) prepareConsulTokensForServices(services []*structs.Service, tg *structs.TaskGroup, tokens map[string]map[string]*consulapi.ACLToken, env *taskenv.TaskEnv) error {
	var mErr *multierror.Error
	for _, service := range services {
		// Exit early if service doesn't need a Consul token.
		if service == nil || !service.IsConsul() || service.Identity == nil {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped error in the agent log to identify whether it's connectivity, login, or ACL preflight
2. Verify the Consul agent address is reachable from the client and the auth method exists in Consul (consul acl auth-method list)
3. Confirm Consul has the Nomad-managed auth method and binding rules for task identities (requires Consul 1.17+/Nomad 1.7+ compatible versions)
4. Check clock synchronization (NTP) between Nomad client and Consul, then reschedule the allocation

Example fix

// before: auth method mismatch
// after: align Nomad client consul config with Consul auth method
consul {
  cluster = "default"
  # ensure service_identity auth method matches the one created in Consul
}

// pre-check Consul reachability and auth method before scheduling
resp, err := consulClient.ACL().AuthMethodList(nil)
if err != nil { return fmt.Errorf("consul unreachable: %w", err) }
// ensure TaskIdentityAuthMethod is present in resp

if err := hook.Prerun(); err != nil {
    if strings.Contains(err.Error(), "failed to derive Consul token") {
        // inspect wrapped cause; retry with backoff for transient network errors
        // do not retry if auth-method mismatch: fix Consul config instead
    }
}

When it happens

Trigger: Consul agent unreachable/wrong address; consulConfig.TaskIdentityAuthMethod not configured or mismatched with the Consul auth method; Consul rejects the JWT (expired signature, audience mismatch); derived token fails the preflight ACL check.

Common situations: Consul agent down or bound to a different interface; auth method deleted/renamed in Consul; Consul and Nomad versions mismatched on the workload-identity auth method; clock skew invalidating the JWT; Consul ACLs enabled without the bound-roles setup.

Related errors

• failed to derive Consul token for service %s: %v
• [✘] Could not create Consul auth method: %w
• no one-time token returned
• no ACL token returned
• errMissingACLRoleID

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/04baa92e0bbff164. Report an issue: GitHub.