{"record":{"id":"00b3b85f46a51a4c","repo":"hashicorp/nomad","slug":"failed-to-resolve-acl-token-v","errorCode":null,"errorMessage":"failed to resolve ACL token: %v","messagePattern":"failed to resolve ACL token: (.+?)","errorType":"http","errorClass":null,"httpStatus":400,"severity":"error","filePath":"command/agent/http.go","lineNumber":432,"sourceCode":"\tvar err error\n\n\tif srv := s.agent.Server(); srv != nil {\n\t\tr := &structs.GenericRequest{}\n\t\tr.AuthToken = secret\n\t\tif authErr := srv.Authenticate(nil, r); authErr != nil {\n\t\t\treturn nil, fmt.Errorf(\"ACL token not found or invalid workload identity: %v\", authErr)\n\t\t}\n\n\t\taclObj, err = srv.ResolveACL(r)\n\t} else {\n\t\t// Not a Server, so use the Client for token resolution. Note\n\t\t// this gets forwarded to a server with AllowStale = true if\n\t\t// the local ACL cache TTL has expired (30s by default)\n\t\taclObj, err = s.agent.Client().ResolveToken(secret)\n\t}\n\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to resolve ACL token: %v\", err)\n\t}\n\n\treturn aclObj, nil\n}\n\n// registerHandlers is used to attach our handlers to the mux\nfunc (s *HTTPServer) registerHandlers(enableDebug bool) {\n\ts.mux.HandleFunc(\"/v1/jobs\", s.wrap(s.JobsRequest))\n\ts.mux.HandleFunc(\"/v1/jobs/parse\", s.wrap(s.JobsParseRequest))\n\ts.mux.HandleFunc(\"/v1/jobs/statuses\", s.wrap(s.JobStatusesRequest))\n\ts.mux.HandleFunc(\"/v1/job/\", s.wrap(s.JobSpecificRequest))\n\n\ts.mux.HandleFunc(\"/v1/nodes\", s.wrap(s.NodesRequest))\n\ts.mux.HandleFunc(\"/v1/node/\", s.wrap(s.NodeSpecificRequest))\n\n\ts.mux.HandleFunc(\"/v1/node/pools\", s.wrap(s.NodePoolsRequest))\n\ts.mux.HandleFunc(\"/v1/node/pool/\", s.wrap(s.NodePoolSpecificRequest))\n","sourceCodeStart":414,"sourceCodeEnd":450,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/command/agent/http.go#L414-L450","documentation":"Returned by ResolveToken (command/agent/http.go:432) when ACL object resolution fails after authentication — either the server path's srv.ResolveACL errored, or the client path's Client().ResolveToken (forwarded to a server with AllowStale, 30s ACL cache TTL) returned an error. It wraps the underlying resolution error.","triggerScenarios":"Requests that authenticate but fail resolution: server-side ResolveACL error, or a client agent that cannot reach any server for forwarded token resolution (network partition, no reachable servers, RPC failure) while ACLs are enabled.","commonSituations":"Client agents with unreachable/misconfigured `servers`; cluster partitions or upgrade windows; stale ACL cache combined with server unavailability; inconsistent ACL state after bootstrap issues.","solutions":["Verify the agent can reach its servers: check `servers` config, retry_join, and connectivity on port 4647.","Confirm cluster health (`nomad server members`, `nomad node status`) and retry the request.","Validate the token with `nomad acl token self`; create a new one if invalid.","Restart the client agent to re-establish server connections and refresh the ACL cache.","Check server logs for ResolveACL/RPC errors to find the root cause."],"exampleFix":"// before: client pointing at an unreachable server (HCL)\nclient {\n  servers = [\"10.0.0.99:4647\"]\n}\n// after\nclient {\n  servers = [\"10.0.0.1:4647\", \"10.0.0.2:4647\", \"10.0.0.3:4647\"]\n}","handlingStrategy":"retry","validationCode":"for _, srv := range cfg.Client.Servers {\n    c, err := net.DialTimeout(\"tcp\", srv, 2*time.Second)\n    if err != nil {\n        return fmt.Errorf(\"server unreachable: %s\", srv)\n    }\n    c.Close()\n}","typeGuard":null,"tryCatchPattern":"if err != nil {\n    if strings.Contains(err.Error(), \"failed to resolve ACL token\") {\n        // transient RPC failure: retry with backoff; if persistent, restart the client agent\n    }\n}","preventionTips":["Configure multiple servers in the client's servers list.","Monitor client-to-server connectivity on port 4647.","Keep the ACL cache TTL default (30s) so AllowStale forwarding covers brief partitions.","Restart client agents after prolonged partitions to refresh ACL state."],"tags":["acl","token-resolution","rpc","nomad"],"backgroundTag":"acl-token-resolution-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}