hashicorp/nomad · error

node update failed: %v

Error message

node update failed: %v

What it means

upsertNodeEvents wraps a failure from txn.Insert("nodes", copyNode) when re-inserting the mutated node record. This indicates the in-transaction write failed (memdb/bolt insert error), not that the node is missing. Raised through StateStore.UpsertNodeEvents.

Source

Thrown at nomad/state/state_store.go:1383

// older events once this bound has been reached.
func (s *StateStore) upsertNodeEvents(index uint64, nodeID string, events []*structs.NodeEvent, txn *txn) error {
	// Lookup the node
	existing, err := txn.First("nodes", "id", nodeID)
	if err != nil {
		return fmt.Errorf("node lookup failed: %v", err)
	}
	if existing == nil {
		return fmt.Errorf("node not found")
	}

	// Copy the existing node
	existingNode := existing.(*structs.Node)
	copyNode := existingNode.Copy()
	appendNodeEvents(index, copyNode, events)

	// Insert the node
	if err := txn.Insert("nodes", copyNode); err != nil {
		return fmt.Errorf("node update failed: %v", err)
	}
	if err := txn.Insert("index", &IndexEntry{"nodes", index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}

	return nil
}

// appendNodeEvents is a helper that takes a node and new events and appends
// them, pruning older events as needed.
func appendNodeEvents(index uint64, node *structs.Node, events []*structs.NodeEvent) {
	// Add the events, updating the indexes
	for _, e := range events {
		e.CreateIndex = index
		node.Events = append(node.Events, e)
	}

	// Keep node events pruned to not exceed the max allowed

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped error (%v) for the underlying memdb/bolt cause.
2. Retry the operation after the transaction is healthy; ensure the txn is not aborted before this insert.
3. Restart the server agent / restore from snapshot if the state store is corrupted.
4. Check disk space and file permissions on the Nomad data directory.

Example fix

// before
if err := txn.Insert("nodes", copyNode); err != nil {
    return fmt.Errorf("node update failed: %v", err)
}
// after
if err := txn.Insert("nodes", copyNode); err != nil {
    return fmt.Errorf("node update failed (node %s): %w", nodeID, err)
}

node, err := store.NodeByID(nil, nodeID)
if err != nil {
    return err
}
if node == nil {
    return nil
}

err := store.UpsertNodeEvents(idx, nodeID, events)
for i := 0; i < 3 && err != nil && strings.HasPrefix(err.Error(), "node update failed"); i++ {
    time.Sleep(backoff)
    err = store.UpsertNodeEvents(idx, nodeID, events)
}

When it happens

Trigger: txn.Insert fails during a node event update — caused by underlying store errors, transaction state issues, or attempting to insert into an aborted/invalid transaction within a composite write (e.g. inside UpdateNodeStatus/Drain transactions).

Common situations: Disk/full or corrupted bolt backend during event-heavy workloads; a nested transaction abort earlier in the call chain making subsequent inserts fail; internal bugs in the write path.

Related errors

• node lookup failed: %v
• failed to retrieve jobs for idempotency check
• failed to lookup state snapshot: %v
• volume update failed: %s: %v
• failed inserting variable: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/a2804b9ceaf9b881. Report an issue: GitHub.