{"record":{"id":"40b611716b45cbfd","repo":"hashicorp/nomad","slug":"acl-auth-method-lookup-failed-v","errorCode":null,"errorMessage":"ACL auth method lookup failed: %v","messagePattern":"ACL auth method lookup failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store_acl_binding_rule.go","lineNumber":74,"sourceCode":"// caller to update the index table.\nfunc (s *StateStore) upsertACLBindingRuleTxn(\n\tindex uint64, txn *txn, rule *structs.ACLBindingRule, allowMissingAuthMethod bool) (bool, error) {\n\n\t// Ensure the rule hash is not zero to provide defense in depth. This\n\t// should be done outside the state store, so we do not spend time here and\n\t// thus Raft, when it can be avoided.\n\tif len(rule.Hash) == 0 {\n\t\trule.SetHash()\n\t}\n\n\t// This validation also happens within the RPC handler, but Raft latency\n\t// could mean that by the time the state call is invoked, another Raft\n\t// update has the auth method detailed in binding rule. Therefore, check\n\t// again while in our write txn.\n\tif !allowMissingAuthMethod {\n\t\tmethod, err := s.GetACLAuthMethodByName(nil, rule.AuthMethod)\n\t\tif err != nil {\n\t\t\treturn false, fmt.Errorf(\"ACL auth method lookup failed: %v\", err)\n\t\t}\n\t\tif method == nil {\n\t\t\treturn false, fmt.Errorf(\"ACL binding rule insert failed: ACL auth method not found\")\n\t\t}\n\t}\n\n\t// This validation also happens within the RPC handler, but Raft latency\n\t// could mean that by the time the state call is invoked, another Raft\n\t// update has already written a method with the same name. We therefore\n\t// need to check we are not trying to create a rule with an existing ID.\n\texistingRaw, err := txn.First(TableACLBindingRules, indexID, rule.ID)\n\tif err != nil {\n\t\treturn false, fmt.Errorf(\"ACL binding rule lookup failed: %v\", err)\n\t}\n\n\tvar existing *structs.ACLBindingRule\n\tif existingRaw != nil {\n\t\texisting = existingRaw.(*structs.ACLBindingRule)","sourceCodeStart":56,"sourceCodeEnd":92,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store_acl_binding_rule.go#L56-L92","documentation":"Returned by upsertACLBindingRuleTxn when re-validating inside the write transaction that the binding rule's referenced AuthMethod exists, and that lookup itself errors (as opposed to returning nil). The error from GetACLAuthMethodByName is wrapped.","triggerScenarios":"UpsertACLBindingRules applied via Raft while GetACLAuthMethodByName fails internally (state store issue) and allowMissingAuthMethod is false.","commonSituations":"State store corruption or version skew during an upgrade; simultaneous Raft operations causing transient internal issues.","solutions":["Inspect the wrapped underlying error to find the lookup root cause.","Retry the upsert once state is consistent.","Ensure all servers run the same Nomad version; restore from snapshot if corrupted."],"exampleFix":"// before\nerr := upsertACLBindingRule(rule) // fails if auth method row is unreadable\n// after\nif _, err := client.ACLAuthMethods().Get(rule.AuthMethod); err != nil {\n    return fmt.Errorf(\"auth method %q unavailable: %w\", rule.AuthMethod, err)\n}\nupsertACLBindingRule(rule)","handlingStrategy":"retry","validationCode":"m, err := client.ACLAuthMethods().Get(rule.AuthMethod)\nif err != nil { return err }\nif m == nil { return fmt.Errorf(\"auth method %q must exist before rule upsert\", rule.AuthMethod) }","typeGuard":null,"tryCatchPattern":"err := upsertRule(rule)\nif err != nil && strings.Contains(err.Error(), \"ACL auth method lookup failed\") {\n    retryWithBackoff(upsertRule, rule)\n}","preventionTips":["Create auth methods before their binding rules","Serialize automation to avoid racing deletes","Check server logs for state-store issues"],"tags":["nomad","state-store","acl","binding-rule","auth-method"],"backgroundTag":"state-store-lookup-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}