hashicorp/nomad · error

upserting policy failed: %v

Error message

upserting policy failed: %v

What it means

Thrown when txn.Insert("acl_policy", policy) fails while upserting an ACL policy into the state store. The insert is the final write of the policy object into the acl_policy table; a failure aborts the transaction so no policy change is persisted.

Source

Thrown at nomad/state/state_store.go:6273

		// Check if the policy already exists
		existing, err := txn.First("acl_policy", "id", policy.Name)
		if err != nil {
			return fmt.Errorf("policy lookup failed: %v", err)
		}

		// Update all the indexes
		if existing != nil {
			policy.CreateIndex = existing.(*structs.ACLPolicy).CreateIndex
			policy.ModifyIndex = index
		} else {
			policy.CreateIndex = index
			policy.ModifyIndex = index
		}

		// Update the policy
		if err := txn.Insert("acl_policy", policy); err != nil {
			return fmt.Errorf("upserting policy failed: %v", err)
		}
	}

	// Update the indexes tabl
	if err := txn.Insert("index", &IndexEntry{"acl_policy", index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}

	return txn.Commit()
}

// DeleteACLPolicies deletes the policies with the given names
func (s *StateStore) DeleteACLPolicies(msgType structs.MessageType, index uint64, names []string) error {
	txn := s.db.WriteTxnMsgT(msgType, index)
	defer txn.Abort()

	// Delete the policy
	for _, name := range names {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry the policy upsert request
2. Verify server health (memory, raft state) before bulk-applying many policies
3. Restore from a consistent snapshot if corruption is indicated in logs
4. Upgrade Nomad and report persistent reproducibility to HashiCorp

// Validate the policy object before submitting
function isValidPolicy(p) {
  return typeof p.Name === 'string' && p.Name.length > 0 &&
         typeof p.Rules === 'string' && p.Rules.length > 0;
}

function isACLPolicy(p) {
  return p !== null && typeof p === 'object' && typeof p.Name === 'string' && typeof p.Rules === 'string';
}

try {
  await nomad.put(`/v1/acl/policy/${encodeURIComponent(p.Name)}`, p);
} catch (e) {
  if (String(e).includes('upserting policy failed') && attempt < 3) return retry(op);
  throw e;
}

When it happens

Trigger: UpsertACLPolicies processes a policy whose hash was set and existing row (if any) reconciled, then the memdb Insert of the ACLPolicy object errors — internal memdb failure, memory exhaustion, or store corruption.

Common situations: Bulk policy upload via nomd acl policy apply / API in a loop hitting a degraded server; memory-constrained servers; corrupted state after failed snapshot restore.

Related errors

• policy lookup failed: %v
• token lookup failed: %v
• acl token lookup failed: %v
• ACL role insert failed: %v
• ACL policy lookup failed: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/5c8c9129b64d3aca. Report an issue: GitHub.