{"record":{"id":"703b9772bf1ad542","repo":"hashicorp/nomad","slug":"index-update-failed-v-703b97","errorCode":null,"errorMessage":"index update failed: %v","messagePattern":"index update failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store_acl_sso.go","lineNumber":47,"sourceCode":"\tfor _, method := range aclAuthMethods {\n\n\t\tmethodUpdated, err := s.upsertACLAuthMethodTxn(index, txn, method)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\n\t\t// Ensure we track whether any inserts have been made.\n\t\tupdated = updated || methodUpdated\n\t}\n\n\t// If we did not perform any inserts, exit early.\n\tif !updated {\n\t\treturn nil\n\t}\n\n\t// Perform the index table update to mark the new insert.\n\tif err := txn.Insert(tableIndex, &IndexEntry{TableACLAuthMethods, index}); err != nil {\n\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t}\n\n\treturn txn.Commit()\n}\n\n// upsertACLAuthMethodTxn inserts a single ACL auth method into the state store\n// using the provided write transaction. It is the responsibility of the caller\n// to update the index table.\nfunc (s *StateStore) upsertACLAuthMethodTxn(index uint64, txn *txn, method *structs.ACLAuthMethod) (bool, error) {\n\n\t// Ensure the method hash is not zero to provide defense in depth. This\n\t// should be done outside the state store, so we do not spend time here and\n\t// thus Raft, when it can be avoided.\n\tif len(method.Hash) == 0 {\n\t\tmethod.SetHash()\n\t}\n\n\t// This validation also happens within the RPC handler, but Raft latency","sourceCodeStart":29,"sourceCodeEnd":65,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store_acl_sso.go#L29-L65","documentation":"UpsertACLAuthMethods finishes by recording the new Raft index in the state store's index table for TableACLAuthMethods; this error wraps a failure of that bookkeeping insert. Because the transaction is only committed after this step, a failure here rolls back the entire auth-method upsert.","triggerScenarios":"Calling StateStore.UpsertACLAuthMethods (ACL auth-method create/update RPC, CLI commands like nomad acl auth-method create) where the index-table txn.Insert fails after the method rows were written.","commonSituations":"State store index-table corruption after crash/restore; zero/invalid index applied from a malformed Raft entry; mixed Nomad versions writing different index entry shapes.","solutions":["Read the wrapped %v cause; restore the state store from a snapshot if it indicates memdb/table corruption.","Retry the auth-method upsert once the cluster has a stable leader.","Check Raft health (nomad operator raft list-peers) and server version consistency.","Escalate with logs if persistent — this internal write should not fail for valid indexes."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"if idx == 0 {\n    return fmt.Errorf(\"invalid raft index for auth method upsert\")\n}","typeGuard":null,"tryCatchPattern":"if err := state.UpsertACLAuthMethods(idx, methods); err != nil {\n    if strings.Contains(err.Error(), \"index update failed\") {\n        // upsert rolled back; wait for stable leader and retry\n        return retryWithBackoff(func() error { return state.UpsertACLAuthMethods(idx, methods) })\n    }\n    return err\n}","preventionTips":["Retry upserts with backoff around leadership changes.","Check Raft health before bulk ACL auth-method changes.","Snapshot the store before large ACL mutations.","Monitor for repeated index-table failures as a corruption indicator."],"tags":["nomad","state-store","acl","index"],"backgroundTag":"state-store-transaction-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}