{"record":{"id":"759e1aea789e0f9a","repo":"hashicorp/nomad","slug":"token-lookup-failed-v","errorCode":null,"errorMessage":"token lookup failed: %v","messagePattern":"token lookup failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":6385,"sourceCode":"\treturn iter, nil\n}\n\n// UpsertACLTokens is used to create or update a set of ACL tokens\nfunc (s *StateStore) UpsertACLTokens(msgType structs.MessageType, index uint64, tokens []*structs.ACLToken) error {\n\ttxn := s.db.WriteTxnMsgT(msgType, index)\n\tdefer txn.Abort()\n\n\tfor _, token := range tokens {\n\t\t// Ensure the policy hash is non-nil. This should be done outside the state store\n\t\t// for performance reasons, but we check here for defense in depth.\n\t\tif len(token.Hash) == 0 {\n\t\t\ttoken.SetHash()\n\t\t}\n\n\t\t// Check if the token already exists\n\t\texisting, err := txn.First(\"acl_token\", \"id\", token.AccessorID)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"token lookup failed: %v\", err)\n\t\t}\n\n\t\t// Update all the indexes\n\t\tif existing != nil {\n\t\t\texistTK := existing.(*structs.ACLToken)\n\t\t\ttoken.CreateIndex = existTK.CreateIndex\n\t\t\ttoken.ModifyIndex = index\n\n\t\t\t// Do not allow SecretID or create time to change\n\t\t\ttoken.SecretID = existTK.SecretID\n\t\t\ttoken.CreateTime = existTK.CreateTime\n\n\t\t} else {\n\t\t\ttoken.CreateIndex = index\n\t\t\ttoken.ModifyIndex = index\n\t\t}\n\n\t\t// Update the token","sourceCodeStart":6367,"sourceCodeEnd":6403,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L6367-L6403","documentation":"This error occurs inside StateStore.UpsertACLTokens when txn.First on the 'acl_token' table by accessor ID fails. It means the memdb read of the existing token errored (index/table problem), aborting the whole token upsert transaction. The upsert transaction is rolled back and the Raft apply fails.","triggerScenarios":"UpsertACLTokens (e.g., an ACLTokenUpsertRequest via the state store) hitting a memdb error on 'acl_token'/'id' — schema mismatch, nil txn, or corrupted in-memory table; concurrent store shutdown mid-write.","commonSituations":"State store schema drift after an upgrade; a forked Nomad with modified acl_token indexes; tests using a closed or mis-initialized StateStore.","solutions":["Restart the Nomad server to reinitialize the in-memory state store from BoltDB.","Verify the acl_token table schema (index 'id') matches the running binary; align versions.","Capture the wrapped %v cause in logs; if it indicates corruption, restore from the latest state snapshot backup."],"exampleFix":"// before\nexisting, err := txn.First(\"acl_token\", \"id\", token.AccessorID)\nif err != nil {\n    return fmt.Errorf(\"token lookup failed: %v\", err)\n}\n// after\nexisting, err := txn.First(\"acl_token\", \"id\", token.AccessorID)\nif err != nil {\n    return fmt.Errorf(\"token lookup failed: %w\", err)\n}","handlingStrategy":"try-catch","validationCode":"func validTokenForUpsert(t *structs.ACLToken) bool {\n    return t != nil && t.AccessorID != \"\" && t.SecretID != \"\"\n}","typeGuard":null,"tryCatchPattern":"if err := store.UpsertACLTokens(msgType, index, tokens); err != nil {\n    if strings.Contains(err.Error(), \"token lookup failed\") {\n        return fmt.Errorf(\"state store rejected token upsert (schema/store issue): %w\", err)\n    }\n    return err\n}","preventionTips":["Call token.SetHash() and ensure AccessorID/SecretID before upsert","Restart servers cleanly on state store errors rather than retrying blindly","Keep acl_token schema identical to the running binary version","Monitor server logs for repeated wrapped memdb errors"],"tags":["nomad","state-store","memdb","acl","raft"],"backgroundTag":"state-store-write-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}