What causes "deleting acl policy failed: %v" in hashicorp/nomad?

Retry the delete; the aborted transaction left state consistent

deleting acl policy failed: %v

Error message

deleting acl policy failed: %v

What it means

Thrown by DeleteACLPolicies when txn.DeleteAll("acl_policy", "id", name) fails for one of the requested policy names. The delete of the policy row(s) failed inside memdb, so the transaction aborts and none of the named policies are removed.

Source

Thrown at nomad/state/state_store.go:6293

	}

	// Update the indexes tabl
	if err := txn.Insert("index", &IndexEntry{"acl_policy", index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}

	return txn.Commit()
}

// DeleteACLPolicies deletes the policies with the given names
func (s *StateStore) DeleteACLPolicies(msgType structs.MessageType, index uint64, names []string) error {
	txn := s.db.WriteTxnMsgT(msgType, index)
	defer txn.Abort()

	// Delete the policy
	for _, name := range names {
		if _, err := txn.DeleteAll("acl_policy", "id", name); err != nil {
			return fmt.Errorf("deleting acl policy failed: %v", err)
		}
	}
	if err := txn.Insert("index", &IndexEntry{"acl_policy", index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}
	return txn.Commit()
}

// ACLPolicyByName is used to lookup a policy by name
func (s *StateStore) ACLPolicyByName(ws memdb.WatchSet, name string) (*structs.ACLPolicy, error) {
	txn := s.db.ReadTxn()

	watchCh, existing, err := txn.FirstWatch("acl_policy", "id", name)
	if err != nil {
		return nil, fmt.Errorf("acl policy lookup failed: %v", err)
	}
	ws.Add(watchCh)

View on GitHub (pinned to 482b49bf1a)

Solutions

Retry the delete; the aborted transaction left state consistent
Delete policies one at a time to isolate a problematic name/table row
Check server logs for memdb errors and resource issues
Restore from snapshot if the acl_policy table appears corrupt

Defensive patterns

Strategy: retry

Validate before calling

// Check the policy exists before deleting (optional; missing names do not error)
const list = await nomad.get('/v1/acl/policies');
const missing = names.filter(n => !list.some(p => p.Name === n));
console.warn('not found (skipped):', missing);

Try / catch

try {
  await Promise.all(names.map(n => nomad.del(`/v1/acl/policy/${encodeURIComponent(n)}`)));
} catch (e) {
  if (String(e).includes('deleting acl policy failed')) return retryWithBackoff(deleteAll, names);
  throw e;
}

Prevention

Delete policies in smaller batches to reduce blast radius and ease retries
Retry is safe — the transaction aborts atomically
Check server logs/mem health if deletes repeatedly fail
Snapshot state before bulk ACL cleanup

When it happens

Trigger: Client issues ACL policy deletion (API/CLI nomad acl policy delete, or batch delete) and the underlying memdb DeleteAll errors — internal store failure rather than 'policy not found' (missing policies simply delete zero rows).

Common situations: Deleting many policies in a batch against a degraded server; corrupted acl_policy table; memory exhaustion on the Nomad server.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/0dc6d60143c08f33. Report an issue: GitHub.