{"record":{"id":"4dea2f889cd702c4","repo":"hashicorp/nomad","slug":"failed-to-derive-consul-token-for-service-s-v","errorCode":null,"errorMessage":"failed to derive Consul token for service %s: %v","messagePattern":"failed to derive Consul token for service (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/allocrunner/consul_hook.go","lineNumber":235,"sourceCode":"\t\t\t\t\t\"error getting signed identity for service %s: %v\",\n\t\t\t\t\tservice.Name, err,\n\t\t\t\t))\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\th.logger.Debug(\"logging into consul\", \"name\", ti.IdentityName, \"type\", ti.WorkloadType)\n\t\t\treq := consul.JWTLoginRequest{\n\t\t\t\tJWT:            swi.JWT,\n\t\t\t\tAuthMethodName: consulConfig.ServiceIdentityAuthMethod,\n\t\t\t\tMeta: map[string]string{\n\t\t\t\t\t\"requested_by\": fmt.Sprintf(\"nomad_service_%s\", ti.InterpolatedWorkloadIdentifier),\n\t\t\t\t\t\"node_id\":      h.alloc.NodeID,\n\t\t\t\t},\n\t\t\t}\n\n\t\t\ttoken, err = h.getConsulToken(clusterName, req)\n\t\t\tif err != nil {\n\t\t\t\tmErr = multierror.Append(mErr, fmt.Errorf(\n\t\t\t\t\t\"failed to derive Consul token for service %s: %v\",\n\t\t\t\t\tservice.Name, err,\n\t\t\t\t))\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t}\n\n\t\t// Store token in results.\n\t\tif _, ok = tokens[clusterName]; !ok {\n\t\t\ttokens[clusterName] = make(map[string]*consulapi.ACLToken)\n\t\t}\n\n\t\ttokens[clusterName][tokenName] = token\n\t}\n\n\treturn mErr.ErrorOrNil()\n}","sourceCodeStart":217,"sourceCodeEnd":253,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/allocrunner/consul_hook.go#L217-L253","documentation":"After obtaining the signed service identity, the hook calls getConsulToken to perform the Consul JWT login (DeriveTokenWithJWT) and a token preflight ACL check. Failures are appended to the multierror as 'failed to derive Consul token for service %s'. The wrapped inner error carries the real cause: connectivity, auth method, JWT validity, or ACL scope.","triggerScenarios":"Consul agent unreachable; ServiceIdentityAuthMethod missing/mismatched in Consul; Consul rejects the service JWT (expired, wrong audience); derived token fails TokenPreflightCheck due to missing ACL permissions.","commonSituations":"Consul auth method or binding rules for service identities not bootstrapped; Consul agent restarted with ACLs reset; network partition between Nomad client and Consul; version incompatibility between Nomad's expected auth method and Consul's.","solutions":["Read the wrapped inner error from the agent log to classify connectivity vs login vs ACL failure","Ensure the Consul auth method and binding rules for Nomad service identities exist (consul acl auth-method list / binding-rule list)","Test connectivity from the client to the Consul agent address/port; fix firewalls or consul.address config","Reschedule the allocation after fixing; check NTP clock sync so JWTs are not rejected as expired"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// pre-validate auth method for service identities on the target Consul\nmethods, _, err := consulClient.ACL().AuthMethodList(nil)\nif err != nil { return err }\nfound := false\nfor _, m := range methods {\n    if m.Name == consulConfig.ServiceIdentityAuthMethod { found = true }\n}\nif !found { return fmt.Errorf(\"auth method %q missing in Consul\", consulConfig.ServiceIdentityAuthMethod) }","typeGuard":null,"tryCatchPattern":"if err := hook.Prerun(); err != nil {\n    if strings.Contains(err.Error(), \"failed to derive Consul token for service\") {\n        // parse wrapped cause; retry transient network failures with backoff,\n        // fix Consul ACL/auth-method config for permanent ones\n    }\n}","preventionTips":["Bootstrap service-identity auth methods and binding rules on every Consul cluster used","Monitor Consul agent health from Nomad clients (consul members/health checks)","Keep Nomad and Consul versions aligned on the JWT auth method API","Enable NTP on both Nomad clients and Consul servers"],"tags":["consul","jwt-login","acl","services","network"],"backgroundTag":"consul-jwt-login-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}