hashicorp/nomad · error

ACL binding rule insert failed: %v

Error message

ACL binding rule insert failed: %v

What it means

Nomad's state store failed to insert an ACL binding rule row into the memdb table during a write transaction (txn.Insert on TableACLBindingRules returned an error). This wraps a low-level memdb error, which in practice almost always means a schema/table definition problem or a corrupted/invalid index rather than a caller-facing condition. It is surfaced from upsertACLBindingRuleTxn, which runs inside UpsertACLBindingRules and is also exercised by event-stream tests.

Source

Thrown at nomad/state/state_store_acl_binding_rule.go:117

		// If the rule already exists, check whether the update contains any
		// difference. If it doesn't, we can avoid a state update as well as
		// updates to any blocking queries.
		if existing.Equal(rule) {
			return false, nil
		}

		rule.CreateIndex = existing.CreateIndex
		rule.ModifyIndex = index
		rule.CreateTime = existing.CreateTime
	} else {
		rule.CreateIndex = index
		rule.ModifyIndex = index
	}

	// Insert the auth method into the table.
	if err := txn.Insert(TableACLBindingRules, rule); err != nil {
		return false, fmt.Errorf("ACL binding rule insert failed: %v", err)
	}
	return true, nil
}

// DeleteACLBindingRules is responsible for batch deleting ACL binding rules.
// It uses a single write transaction for efficiency, however, any error means
// no entries will be committed. An error is produced if a rule is not found
// within state which has been passed within the array.
func (s *StateStore) DeleteACLBindingRules(index uint64, bindingRuleIDs []string) error {
	txn := s.db.WriteTxnMsgT(structs.ACLBindingRulesDeleteRequestType, index)
	defer txn.Abort()

	for _, ruleID := range bindingRuleIDs {
		if err := s.deleteACLBindingRuleTxn(txn, ruleID); err != nil {
			return err
		}
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped %v cause in the log to identify the underlying memdb error and fix the offending object (e.g. ensure the binding rule has a valid ID and AuthMethod).
2. If it appears after an unclean shutdown or restore, restore the state store from a snapshot (nomad snapshot inspect/save) or re-run with a rebuilt Raft state.
3. Verify Nomad version consistency across agents/clients; upgrade to a release with the ACL binding rule schema fixes if you recently upgraded.
4. If reproducible, file an issue with the wrapped error text — this path should never fail for well-formed inputs.

Example fix

// before: rule constructed without ID
rule := &structs.ACLBindingRule{AuthMethod: "auth0"}
// after
rule := &structs.ACLBindingRule{ID: uuid.Generate(), AuthMethod: "auth0"}

if rule == nil || rule.ID == "" || rule.AuthMethod == "" {
    return fmt.Errorf("invalid ACL binding rule: missing ID or AuthMethod")
}

func isValidBindingRule(r *structs.ACLBindingRule) bool {
    return r != nil && r.ID != "" && r.AuthMethod != ""
}

if err := stateStore.UpsertACLBindingRules(idx, rules); err != nil {
    if strings.Contains(err.Error(), "ACL binding rule insert failed") {
        // log wrapped cause, restore/rebuild state store before retrying
        return fmt.Errorf("state store insert rejected: %w", err)
    }
    return err
}

When it happens

Trigger: Calling StateStore.UpsertACLBindingRules (via the ACL binding rule upsert RPC or Restore path) when the underlying memdb txn.Insert against TableACLBindingRules fails — e.g. the binding rule object violates the table's index schema (nil ID, malformed index tuple) or the memdb table definition is inconsistent.

Common situations: Corrupted state store files after an unclean shutdown or a failed restore; internal upgrades that change the ACL binding rule table schema; bugs in code constructing structs.ACLBindingRule with empty/invalid fields before upsert; tests injecting malformed rules via Test_eventsFromChanges_ACLBindingRule.

Related errors

• policy lookup failed: %v
• upserting policy failed: %v
• token lookup failed: %v
• acl token lookup failed: %v
• ACL role insert failed: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/15dc47edd445b5ba. Report an issue: GitHub.