What causes "ACL binding rule deletion failed: %v" in hashicorp/nomad?

Restore the state store from a known-good snapshot taken with the current Nomad version.

ACL binding rule deletion failed: %v

Error message

ACL binding rule deletion failed: %v

What it means

This error means the ACL binding rule row was found but txn.Delete failed to remove it from the memdb table. Like the other wraps here, it signals an internal state store problem, since a successfully located row should always be deletable inside a write transaction.

Source

Thrown at nomad/state/state_store_acl_binding_rule.go:158

	return txn.Commit()
}

// deleteACLBindingRuleTxn deletes a single ACL binding rule from the state
// store using the provided write transaction. It is the responsibility of the
// caller to update the index table.
func (s *StateStore) deleteACLBindingRuleTxn(txn *txn, ruleID string) error {
	existing, err := txn.First(TableACLBindingRules, indexID, ruleID)
	if err != nil {
		return fmt.Errorf("ACL binding rule lookup failed: %v", err)
	}
	if existing == nil {
		return errors.New("ACL binding rule not found")
	}

	// Delete the existing entry from the table.
	if err := txn.Delete(TableACLBindingRules, existing); err != nil {
		return fmt.Errorf("ACL binding rule deletion failed: %v", err)
	}
	return nil
}

// GetACLBindingRules returns an iterator that contains all ACL binding rules
// stored within state.
func (s *StateStore) GetACLBindingRules(ws memdb.WatchSet) (memdb.ResultIterator, error) {
	txn := s.db.ReadTxn()

	// Walk the entire table to get all ACL binding rules.
	iter, err := txn.Get(TableACLBindingRules, indexID)
	if err != nil {
		return nil, fmt.Errorf("ACL binding rules lookup failed: %v", err)
	}
	ws.Add(iter.WatchCh())

	return iter, nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

Restore the state store from a known-good snapshot taken with the current Nomad version.
Verify all servers run the same Nomad version; rolling-upgrade mismatches can leave rows with stale index schemas.
Retry the delete after a leader change if it coincided with failover.
Report with the wrapped error if reproducible — deletion of an existing row should not fail.

Defensive patterns

Strategy: try-catch

Validate before calling

if _, err := state.GetACLBindingRule(nil, ruleID); err != nil {
    return err
} // rule existence already verified before this delete runs

Try / catch

if err := deleteBindingRules(ids); err != nil {
    if strings.Contains(err.Error(), "deletion failed") {
        // likely corrupted row; plan snapshot restore
        log.Error("binding rule delete failed", "cause", err)
        return err
    }
    return err
}

Prevention

Take regular nomad snapshot backups for recovery from corrupted rows.
Avoid downgrading servers below the version that wrote the rows.
Verify cluster health after unclean shutdowns before ACL writes.
Escalate reproducible cases with the wrapped error text.

When it happens

Trigger: DeleteACLBindingRules finds the rule (deleteACLBindingRuleTxn passes the lookup and not-found checks) but txn.Delete(TableACLBindingRules, existing) returns an error, e.g. because the stored object no longer matches the table's index definitions (corrupted row).

Common situations: State store rows written by an incompatible Nomad version; corruption after unclean shutdown; restore snapshots taken from a mismatched version.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/54c529032b3dc9d7. Report an issue: GitHub.