{"record":{"id":"04baa92e0bbff164","repo":"hashicorp/nomad","slug":"failed-to-derive-consul-token-for-task-s-v","errorCode":null,"errorMessage":"failed to derive Consul token for task %s: %v","messagePattern":"failed to derive Consul token for task (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/allocrunner/consul_hook.go","lineNumber":179,"sourceCode":"\t\tti := *task.IdentityHandle(wid)\n\t\tswi, err := h.widmgr.Get(ti)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"error getting signed identity for task %s: %v\", task.Name, err)\n\t\t}\n\n\t\th.logger.Debug(\"logging into consul\", \"name\", ti.IdentityName, \"type\", ti.WorkloadType)\n\t\treq := consul.JWTLoginRequest{\n\t\t\tJWT:            swi.JWT,\n\t\t\tAuthMethodName: consulConfig.TaskIdentityAuthMethod,\n\t\t\tMeta: map[string]string{\n\t\t\t\t\"requested_by\": fmt.Sprintf(\"nomad_task_%s\", task.Name),\n\t\t\t\t\"node_id\":      h.alloc.NodeID,\n\t\t\t},\n\t\t}\n\n\t\ttoken, err = h.getConsulToken(consulConfig.Name, req)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"failed to derive Consul token for task %s: %v\", task.Name, err)\n\t\t}\n\t}\n\n\t// Store token in results.\n\tif _, ok = tokens[clusterName]; !ok {\n\t\ttokens[clusterName] = make(map[string]*consulapi.ACLToken)\n\t}\n\n\ttokens[clusterName][tokenName] = token\n\n\treturn nil\n}\n\nfunc (h *consulHook) prepareConsulTokensForServices(services []*structs.Service, tg *structs.TaskGroup, tokens map[string]map[string]*consulapi.ACLToken, env *taskenv.TaskEnv) error {\n\tvar mErr *multierror.Error\n\tfor _, service := range services {\n\t\t// Exit early if service doesn't need a Consul token.\n\t\tif service == nil || !service.IsConsul() || service.Identity == nil {","sourceCodeStart":161,"sourceCodeEnd":197,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/allocrunner/consul_hook.go#L161-L197","documentation":"After building a JWTLoginRequest from the signed identity, the hook calls getConsulToken, which performs JWT login against Consul (DeriveTokenWithJWT) plus a token preflight check. Any failure is wrapped as 'failed to derive Consul token for task %s'. The root cause is in the wrapped error: unreachable Consul, wrong auth method, invalid/expired JWT, or ACL check failure.","triggerScenarios":"Consul agent unreachable/wrong address; consulConfig.TaskIdentityAuthMethod not configured or mismatched with the Consul auth method; Consul rejects the JWT (expired signature, audience mismatch); derived token fails the preflight ACL check.","commonSituations":"Consul agent down or bound to a different interface; auth method deleted/renamed in Consul; Consul and Nomad versions mismatched on the workload-identity auth method; clock skew invalidating the JWT; Consul ACLs enabled without the bound-roles setup.","solutions":["Read the wrapped error in the agent log to identify whether it's connectivity, login, or ACL preflight","Verify the Consul agent address is reachable from the client and the auth method exists in Consul (consul acl auth-method list)","Confirm Consul has the Nomad-managed auth method and binding rules for task identities (requires Consul 1.17+/Nomad 1.7+ compatible versions)","Check clock synchronization (NTP) between Nomad client and Consul, then reschedule the allocation"],"exampleFix":"// before: auth method mismatch\n// after: align Nomad client consul config with Consul auth method\nconsul {\n  cluster = \"default\"\n  # ensure service_identity auth method matches the one created in Consul\n}","handlingStrategy":"retry","validationCode":"// pre-check Consul reachability and auth method before scheduling\nresp, err := consulClient.ACL().AuthMethodList(nil)\nif err != nil { return fmt.Errorf(\"consul unreachable: %w\", err) }\n// ensure TaskIdentityAuthMethod is present in resp","typeGuard":null,"tryCatchPattern":"if err := hook.Prerun(); err != nil {\n    if strings.Contains(err.Error(), \"failed to derive Consul token\") {\n        // inspect wrapped cause; retry with backoff for transient network errors\n        // do not retry if auth-method mismatch: fix Consul config instead\n    }\n}","preventionTips":["Verify Consul agent address/port reachability from every Nomad client","Bootstrap the Nomad auth method and binding rules in Consul (consul acl bootstrap flow)","Keep Nomad/Consul versions compatible for workload identities (Nomad 1.7+, Consul 1.17+)","Synchronize clocks with NTP to avoid JWT expiry rejections","Set conservative token TTLs so preflight checks pass"],"tags":["consul","jwt-login","acl","network"],"backgroundTag":"consul-jwt-login-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}