What causes "ACL auth method lookup failed: %v" in hashicorp/nomad?

Inspect the wrapped underlying error to find the lookup root cause.

ACL auth method lookup failed: %v

Error message

ACL auth method lookup failed: %v

What it means

Returned by upsertACLBindingRuleTxn when re-validating inside the write transaction that the binding rule's referenced AuthMethod exists, and that lookup itself errors (as opposed to returning nil). The error from GetACLAuthMethodByName is wrapped.

Source

Thrown at nomad/state/state_store_acl_binding_rule.go:74

// caller to update the index table.
func (s *StateStore) upsertACLBindingRuleTxn(
	index uint64, txn *txn, rule *structs.ACLBindingRule, allowMissingAuthMethod bool) (bool, error) {

	// Ensure the rule hash is not zero to provide defense in depth. This
	// should be done outside the state store, so we do not spend time here and
	// thus Raft, when it can be avoided.
	if len(rule.Hash) == 0 {
		rule.SetHash()
	}

	// This validation also happens within the RPC handler, but Raft latency
	// could mean that by the time the state call is invoked, another Raft
	// update has the auth method detailed in binding rule. Therefore, check
	// again while in our write txn.
	if !allowMissingAuthMethod {
		method, err := s.GetACLAuthMethodByName(nil, rule.AuthMethod)
		if err != nil {
			return false, fmt.Errorf("ACL auth method lookup failed: %v", err)
		}
		if method == nil {
			return false, fmt.Errorf("ACL binding rule insert failed: ACL auth method not found")
		}
	}

	// This validation also happens within the RPC handler, but Raft latency
	// could mean that by the time the state call is invoked, another Raft
	// update has already written a method with the same name. We therefore
	// need to check we are not trying to create a rule with an existing ID.
	existingRaw, err := txn.First(TableACLBindingRules, indexID, rule.ID)
	if err != nil {
		return false, fmt.Errorf("ACL binding rule lookup failed: %v", err)
	}

	var existing *structs.ACLBindingRule
	if existingRaw != nil {
		existing = existingRaw.(*structs.ACLBindingRule)

View on GitHub (pinned to 482b49bf1a)

Solutions

Inspect the wrapped underlying error to find the lookup root cause.
Retry the upsert once state is consistent.
Ensure all servers run the same Nomad version; restore from snapshot if corrupted.

Example fix

// before
err := upsertACLBindingRule(rule) // fails if auth method row is unreadable
// after
if _, err := client.ACLAuthMethods().Get(rule.AuthMethod); err != nil {
    return fmt.Errorf("auth method %q unavailable: %w", rule.AuthMethod, err)
}
upsertACLBindingRule(rule)

Defensive patterns

Strategy: retry

Validate before calling

m, err := client.ACLAuthMethods().Get(rule.AuthMethod)
if err != nil { return err }
if m == nil { return fmt.Errorf("auth method %q must exist before rule upsert", rule.AuthMethod) }

Try / catch

err := upsertRule(rule)
if err != nil && strings.Contains(err.Error(), "ACL auth method lookup failed") {
    retryWithBackoff(upsertRule, rule)
}

Prevention

Create auth methods before their binding rules
Serialize automation to avoid racing deletes
Check server logs for state-store issues

When it happens

Trigger: UpsertACLBindingRules applied via Raft while GetACLAuthMethodByName fails internally (state store issue) and allowMissingAuthMethod is false.

Common situations: State store corruption or version skew during an upgrade; simultaneous Raft operations causing transient internal issues.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/40b611716b45cbfd. Report an issue: GitHub.