hashicorp/nomad · error

index update failed: %v

Error message

index update failed: %v

What it means

This error wraps a memdb table-insert failure when updating the node's modify index in the "index" table during a node-event upsert transaction in Nomad's state store. It indicates the Raft-replicated state mutation could not record its index entry, so the whole transaction is aborted and the Raft log entry fails. It is an internal consistency failure, not something caused by user input.

Source

Thrown at nomad/state/state_store.go:1386

	existing, err := txn.First("nodes", "id", nodeID)
	if err != nil {
		return fmt.Errorf("node lookup failed: %v", err)
	}
	if existing == nil {
		return fmt.Errorf("node not found")
	}

	// Copy the existing node
	existingNode := existing.(*structs.Node)
	copyNode := existingNode.Copy()
	appendNodeEvents(index, copyNode, events)

	// Insert the node
	if err := txn.Insert("nodes", copyNode); err != nil {
		return fmt.Errorf("node update failed: %v", err)
	}
	if err := txn.Insert("index", &IndexEntry{"nodes", index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}

	return nil
}

// appendNodeEvents is a helper that takes a node and new events and appends
// them, pruning older events as needed.
func appendNodeEvents(index uint64, node *structs.Node, events []*structs.NodeEvent) {
	// Add the events, updating the indexes
	for _, e := range events {
		e.CreateIndex = index
		node.Events = append(node.Events, e)
	}

	// Keep node events pruned to not exceed the max allowed
	if l := len(node.Events); l > structs.MaxRetainedNodeEvents {
		delta := l - structs.MaxRetainedNodeEvents
		node.Events = node.Events[delta:]

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry the operation — Raft apply failures are often transient; the client re-registers node events on the next heartbeat.
2. Check server logs for an earlier 'node update failed' error in the same transaction; fix the root cause there.
3. Verify the Nomad version for known memdb/raft bugs and upgrade to the latest patch release.
4. If persistent, restart the server agent and, if Raft state is corrupt, restore from a backup as documented in Nomad's recovery guide.

Example fix

// before: error surfaces from an internal txn
// nomad/state/state_store.go (maintainer fix path)
if err := txn.Insert("index", &IndexEntry{"nodes", index}); err != nil {
	return fmt.Errorf("index update failed: %v", err)
}
// after: caller side — treat as transient and retry the RPC
if err := client.Nodes().UpdateStatus(...); err != nil && strings.Contains(err.Error(), "index update failed") {
	time.Sleep(backoff)
	return client.Nodes().UpdateStatus(...)
}

// preflight: ensure the node is registered and server is leader-reachable
_, _, err := client.Nodes().Info(nodeID, nil)
if err != nil { return fmt.Errorf("node unreachable, skip event update: %w", err) }

err := client.Nodes().UpdateStatus(nodeID, "ready", nil)
if err != nil && strings.Contains(err.Error(), "index update failed") {
    // transient raft/state-store write failure: backoff and retry
    time.Sleep(2 * time.Second)
    return retryUpdate(nodeID)
}
return err

When it happens

Trigger: Calling UpsertNodeEvents (via the Node.UpdateStatus / event batching path) when the underlying memdb transaction cannot insert the IndexEntry{TableNodeEvents... "nodes"} row — typically memdb corruption, an invalid table/index name, or transaction already in a failed state after the preceding nodes insert.

Common situations: Seen in operator logs as a failed Raft write when node heartbeat/status updates flush events; usually accompanies memory pressure, disk issues on the Raft layer, or a bug report rather than routine misconfiguration.

Related errors

• alloc %q lookup failed: %v
• token lookup failed: %v
• error querying plugin %q: %v
• index update failed: %v
• JobByID lookup failed: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/dd104c6aefb1f42a. Report an issue: GitHub.