{"record":{"id":"5c8c9129b64d3aca","repo":"hashicorp/nomad","slug":"upserting-policy-failed-v","errorCode":null,"errorMessage":"upserting policy failed: %v","messagePattern":"upserting policy failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":6273,"sourceCode":"\n\t\t// Check if the policy already exists\n\t\texisting, err := txn.First(\"acl_policy\", \"id\", policy.Name)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"policy lookup failed: %v\", err)\n\t\t}\n\n\t\t// Update all the indexes\n\t\tif existing != nil {\n\t\t\tpolicy.CreateIndex = existing.(*structs.ACLPolicy).CreateIndex\n\t\t\tpolicy.ModifyIndex = index\n\t\t} else {\n\t\t\tpolicy.CreateIndex = index\n\t\t\tpolicy.ModifyIndex = index\n\t\t}\n\n\t\t// Update the policy\n\t\tif err := txn.Insert(\"acl_policy\", policy); err != nil {\n\t\t\treturn fmt.Errorf(\"upserting policy failed: %v\", err)\n\t\t}\n\t}\n\n\t// Update the indexes tabl\n\tif err := txn.Insert(\"index\", &IndexEntry{\"acl_policy\", index}); err != nil {\n\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t}\n\n\treturn txn.Commit()\n}\n\n// DeleteACLPolicies deletes the policies with the given names\nfunc (s *StateStore) DeleteACLPolicies(msgType structs.MessageType, index uint64, names []string) error {\n\ttxn := s.db.WriteTxnMsgT(msgType, index)\n\tdefer txn.Abort()\n\n\t// Delete the policy\n\tfor _, name := range names {","sourceCodeStart":6255,"sourceCodeEnd":6291,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L6255-L6291","documentation":"Thrown when txn.Insert(\"acl_policy\", policy) fails while upserting an ACL policy into the state store. The insert is the final write of the policy object into the acl_policy table; a failure aborts the transaction so no policy change is persisted.","triggerScenarios":"UpsertACLPolicies processes a policy whose hash was set and existing row (if any) reconciled, then the memdb Insert of the ACLPolicy object errors — internal memdb failure, memory exhaustion, or store corruption.","commonSituations":"Bulk policy upload via nomd acl policy apply / API in a loop hitting a degraded server; memory-constrained servers; corrupted state after failed snapshot restore.","solutions":["Retry the policy upsert request","Verify server health (memory, raft state) before bulk-applying many policies","Restore from a consistent snapshot if corruption is indicated in logs","Upgrade Nomad and report persistent reproducibility to HashiCorp"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Validate the policy object before submitting\nfunction isValidPolicy(p) {\n  return typeof p.Name === 'string' && p.Name.length > 0 &&\n         typeof p.Rules === 'string' && p.Rules.length > 0;\n}","typeGuard":"function isACLPolicy(p) {\n  return p !== null && typeof p === 'object' && typeof p.Name === 'string' && typeof p.Rules === 'string';\n}","tryCatchPattern":"try {\n  await nomad.put(`/v1/acl/policy/${encodeURIComponent(p.Name)}`, p);\n} catch (e) {\n  if (String(e).includes('upserting policy failed') && attempt < 3) return retry(op);\n  throw e;\n}","preventionTips":["Apply policies in batches with retry/backoff rather than unbounded loops","Monitor server health during bulk ACL changes","Keep Nomad patched; memdb write failures should be rare","Take snapshots before large ACL migrations"],"tags":["nomad","acl","state-store","memdb"],"backgroundTag":"acl-policy-upsert-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}