What causes "index update failed: %v" in hashicorp/nomad?

Read the wrapped %v cause; restore the state store from a snapshot if it indicates memdb/table corruption.

index update failed: %v

Error message

index update failed: %v

What it means

UpsertACLAuthMethods finishes by recording the new Raft index in the state store's index table for TableACLAuthMethods; this error wraps a failure of that bookkeeping insert. Because the transaction is only committed after this step, a failure here rolls back the entire auth-method upsert.

Source

Thrown at nomad/state/state_store_acl_sso.go:47

	for _, method := range aclAuthMethods {

		methodUpdated, err := s.upsertACLAuthMethodTxn(index, txn, method)
		if err != nil {
			return err
		}

		// Ensure we track whether any inserts have been made.
		updated = updated || methodUpdated
	}

	// If we did not perform any inserts, exit early.
	if !updated {
		return nil
	}

	// Perform the index table update to mark the new insert.
	if err := txn.Insert(tableIndex, &IndexEntry{TableACLAuthMethods, index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}

	return txn.Commit()
}

// upsertACLAuthMethodTxn inserts a single ACL auth method into the state store
// using the provided write transaction. It is the responsibility of the caller
// to update the index table.
func (s *StateStore) upsertACLAuthMethodTxn(index uint64, txn *txn, method *structs.ACLAuthMethod) (bool, error) {

	// Ensure the method hash is not zero to provide defense in depth. This
	// should be done outside the state store, so we do not spend time here and
	// thus Raft, when it can be avoided.
	if len(method.Hash) == 0 {
		method.SetHash()
	}

	// This validation also happens within the RPC handler, but Raft latency

View on GitHub (pinned to 482b49bf1a)

Solutions

Read the wrapped %v cause; restore the state store from a snapshot if it indicates memdb/table corruption.
Retry the auth-method upsert once the cluster has a stable leader.
Check Raft health (nomad operator raft list-peers) and server version consistency.
Escalate with logs if persistent — this internal write should not fail for valid indexes.

Defensive patterns

Strategy: retry

Validate before calling

if idx == 0 {
    return fmt.Errorf("invalid raft index for auth method upsert")
}

Try / catch

if err := state.UpsertACLAuthMethods(idx, methods); err != nil {
    if strings.Contains(err.Error(), "index update failed") {
        // upsert rolled back; wait for stable leader and retry
        return retryWithBackoff(func() error { return state.UpsertACLAuthMethods(idx, methods) })
    }
    return err
}

Prevention

Retry upserts with backoff around leadership changes.
Check Raft health before bulk ACL auth-method changes.
Snapshot the store before large ACL mutations.
Monitor for repeated index-table failures as a corruption indicator.

When it happens

Trigger: Calling StateStore.UpsertACLAuthMethods (ACL auth-method create/update RPC, CLI commands like nomad acl auth-method create) where the index-table txn.Insert fails after the method rows were written.

Common situations: State store index-table corruption after crash/restore; zero/invalid index applied from a malformed Raft entry; mixed Nomad versions writing different index entry shapes.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/703b9772bf1ad542. Report an issue: GitHub.