What causes "ACL auth method lookup failed: %v" in hashicorp/nomad?

Retry the upsert; errors during leader election are typically transient.

ACL auth method lookup failed: %v

Error message

ACL auth method lookup failed: %v

What it means

During upsertACLAuthMethodTxn, the store checks whether an auth method with the same name already exists via txn.First(TableACLAuthMethods, "id", method.Name). This error wraps a failure of that existence-check read itself. It aborts the upsert before any insert or update happens.

Source

Thrown at nomad/state/state_store_acl_sso.go:80

		method.SetHash()
	}

	// This validation also happens within the RPC handler, but Raft latency
	// could mean that by the time the state call is invoked, another Raft
	// update has already written a method with the same name or default
	// setting. We therefore need to check we are not trying to create a method
	// with an existing name or a duplicate default for the same type.
	if method.Default {
		existingMethodsDefaultMethod, _ := s.GetDefaultACLAuthMethod(nil)
		if existingMethodsDefaultMethod != nil && existingMethodsDefaultMethod.Name != method.Name {
			return false, fmt.Errorf(
				"default ACL auth method already exists: %v", existingMethodsDefaultMethod.Name,
			)
		}
	}
	existingRaw, err := txn.First(TableACLAuthMethods, indexID, method.Name)
	if err != nil {
		return false, fmt.Errorf("ACL auth method lookup failed: %v", err)
	}

	var existing *structs.ACLAuthMethod
	if existingRaw != nil {
		existing = existingRaw.(*structs.ACLAuthMethod)
	}

	// Depending on whether this is an initial create, or an update, we need to
	// check and set certain parameters. The most important is to ensure any
	// create index is carried over.
	if existing != nil {

		// If the method already exists, check whether the update contains any
		// difference. If it doesn't, we can avoid a state update as well as
		// updates to any blocking queries.
		if existing.Equal(method) {
			return false, nil
		}

View on GitHub (pinned to 482b49bf1a)

Solutions

Retry the upsert; errors during leader election are typically transient.
Inspect the wrapped cause; restore the state store from a verified snapshot if corruption is indicated.
Restart the Nomad server to rebuild memdb from Raft logs.
Ensure all servers run the same Nomad version so table index definitions match.

Defensive patterns

Strategy: retry

Validate before calling

existing, err := state.GetACLAuthMethodByName(nil, method.Name)
if err != nil {
    return retryableError(err)
} // nil existing means name is free; proceed with upsert

Try / catch

if err := upsertAuthMethod(m); err != nil {
    if strings.Contains(err.Error(), "ACL auth method lookup failed") {
        return retryWithBackoff(func() error { return upsertAuthMethod(m) })
    }
    return err
}

Prevention

Retry upserts during leader failover — the read probe can fail transiently.
Verify auth method names are well-formed before upserting.
Restore from snapshot if the wrapped memdb error persists across retries.
Avoid mixed Nomad versions that alter the auth-method table indexes.

When it happens

Trigger: Calling UpsertACLAuthMethods where the duplicate-name existence probe txn.First(TableACLAuthMethods, indexID, method.Name) returns an error rather than a row or nil.

Common situations: Memdb table/index inconsistency after a corrupted restore; version skew changing the auth-method table's 'id' index; transient read failures during leader failover.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/079311e2e4d2be11. Report an issue: GitHub.