What causes "index update failed: %v" in hashicorp/nomad?

Inspect the wrapped error for the memdb root cause.

index update failed: %v

Error message

index update failed: %v

What it means

Returned by UpsertACLBindingRules when inserting the new last-index entry for the acl_binding_rules table into the index table fails. This happens after all rules were written but before txn.Commit(), so the whole write aborts.

Source

Thrown at nomad/state/state_store_acl_binding_rule.go:48

	for _, rule := range bindingRules {

		bindingRuleInserted, err := s.upsertACLBindingRuleTxn(index, txn, rule, allowMissingAuthMethod)
		if err != nil {
			return err
		}

		// Ensure we track whether any inserts have been made.
		updated = updated || bindingRuleInserted
	}

	// If we did not perform any inserts, exit early.
	if !updated {
		return nil
	}

	// Perform the index table update to mark the new insert.
	if err := txn.Insert(tableIndex, &IndexEntry{TableACLBindingRules, index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}

	return txn.Commit()
}

// upsertACLBindingRuleTxn inserts a single ACL binding rule into the state
// store using the provided write transaction. It is the responsibility of the
// caller to update the index table.
func (s *StateStore) upsertACLBindingRuleTxn(
	index uint64, txn *txn, rule *structs.ACLBindingRule, allowMissingAuthMethod bool) (bool, error) {

	// Ensure the rule hash is not zero to provide defense in depth. This
	// should be done outside the state store, so we do not spend time here and
	// thus Raft, when it can be avoided.
	if len(rule.Hash) == 0 {
		rule.SetHash()
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

Inspect the wrapped error for the memdb root cause.
Retry the request; if it repeats, check server logs for state-store corruption.
Restore the cluster state from a snapshot.

Defensive patterns

Strategy: retry

Try / catch

err := upsertBindingRules(rules)
if err != nil && strings.Contains(err.Error(), "index update failed") {
    retryWithBackoff(upsertBindingRules, rules)
}

Prevention

Retry idempotent Raft writes
Keep cluster on one binary version
Restore from snapshot on persistent errors

When it happens

Trigger: txn.Insert(tableIndex, &IndexEntry{TableACLBindingRules, index}) erroring while applying an ACLBindingRulesUpsertRequest via Raft.

Common situations: Corrupted index table in the state store; memdb internal errors; version/schema skew.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/fc21f4045689b7f0. Report an issue: GitHub.