hashicorp/nomad · error

failed to resolve ACL token: %v

Error message

failed to resolve ACL token: %v

What it means

Returned by ResolveToken (command/agent/http.go:432) when ACL object resolution fails after authentication — either the server path's srv.ResolveACL errored, or the client path's Client().ResolveToken (forwarded to a server with AllowStale, 30s ACL cache TTL) returned an error. It wraps the underlying resolution error.

Source

Thrown at command/agent/http.go:432

	var err error

	if srv := s.agent.Server(); srv != nil {
		r := &structs.GenericRequest{}
		r.AuthToken = secret
		if authErr := srv.Authenticate(nil, r); authErr != nil {
			return nil, fmt.Errorf("ACL token not found or invalid workload identity: %v", authErr)
		}

		aclObj, err = srv.ResolveACL(r)
	} else {
		// Not a Server, so use the Client for token resolution. Note
		// this gets forwarded to a server with AllowStale = true if
		// the local ACL cache TTL has expired (30s by default)
		aclObj, err = s.agent.Client().ResolveToken(secret)
	}

	if err != nil {
		return nil, fmt.Errorf("failed to resolve ACL token: %v", err)
	}

	return aclObj, nil
}

// registerHandlers is used to attach our handlers to the mux
func (s *HTTPServer) registerHandlers(enableDebug bool) {
	s.mux.HandleFunc("/v1/jobs", s.wrap(s.JobsRequest))
	s.mux.HandleFunc("/v1/jobs/parse", s.wrap(s.JobsParseRequest))
	s.mux.HandleFunc("/v1/jobs/statuses", s.wrap(s.JobStatusesRequest))
	s.mux.HandleFunc("/v1/job/", s.wrap(s.JobSpecificRequest))

	s.mux.HandleFunc("/v1/nodes", s.wrap(s.NodesRequest))
	s.mux.HandleFunc("/v1/node/", s.wrap(s.NodeSpecificRequest))

	s.mux.HandleFunc("/v1/node/pools", s.wrap(s.NodePoolsRequest))
	s.mux.HandleFunc("/v1/node/pool/", s.wrap(s.NodePoolSpecificRequest))

View on GitHub (pinned to 482b49bf1a)

Solutions

  1. Verify the agent can reach its servers: check `servers` config, retry_join, and connectivity on port 4647.
  2. Confirm cluster health (`nomad server members`, `nomad node status`) and retry the request.
  3. Validate the token with `nomad acl token self`; create a new one if invalid.
  4. Restart the client agent to re-establish server connections and refresh the ACL cache.
  5. Check server logs for ResolveACL/RPC errors to find the root cause.

Example fix

// before: client pointing at an unreachable server (HCL)
client {
  servers = ["10.0.0.99:4647"]
}
// after
client {
  servers = ["10.0.0.1:4647", "10.0.0.2:4647", "10.0.0.3:4647"]
}
Defensive patterns

Strategy: retry

Validate before calling

for _, srv := range cfg.Client.Servers {
    c, err := net.DialTimeout("tcp", srv, 2*time.Second)
    if err != nil {
        return fmt.Errorf("server unreachable: %s", srv)
    }
    c.Close()
}

Try / catch

if err != nil {
    if strings.Contains(err.Error(), "failed to resolve ACL token") {
        // transient RPC failure: retry with backoff; if persistent, restart the client agent
    }
}

Prevention

When it happens

Trigger: Requests that authenticate but fail resolution: server-side ResolveACL error, or a client agent that cannot reach any server for forwarded token resolution (network partition, no reachable servers, RPC failure) while ACLs are enabled.

Common situations: Client agents with unreachable/misconfigured `servers`; cluster partitions or upgrade windows; stale ACL cache combined with server unavailability; inconsistent ACL state after bootstrap issues.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/00b3b85f46a51a4c. Report an issue: GitHub.