hashicorp/nomad · error

node lookup failed: %v

Error message

node lookup failed: %v

What it means

upsertNodeEvents wraps a low-level memdb lookup failure when fetching the node whose events are being appended. Unlike 'node not found' (2518), this indicates the txn.First query itself errored, e.g. an internal store/mdb error. Raised through StateStore.UpsertNodeEvents.

Source

Thrown at nomad/state/state_store.go:1370

	defer txn.Abort()

	for nodeID, events := range nodeEvents {
		if err := s.upsertNodeEvents(index, nodeID, events, txn); err != nil {
			return err
		}
	}

	return txn.Commit()
}

// upsertNodeEvent upserts a node event for a respective node. It also maintains
// that a fixed number of node events are ever stored simultaneously, deleting
// older events once this bound has been reached.
func (s *StateStore) upsertNodeEvents(index uint64, nodeID string, events []*structs.NodeEvent, txn *txn) error {
	// Lookup the node
	existing, err := txn.First("nodes", "id", nodeID)
	if err != nil {
		return fmt.Errorf("node lookup failed: %v", err)
	}
	if existing == nil {
		return fmt.Errorf("node not found")
	}

	// Copy the existing node
	existingNode := existing.(*structs.Node)
	copyNode := existingNode.Copy()
	appendNodeEvents(index, copyNode, events)

	// Insert the node
	if err := txn.Insert("nodes", copyNode); err != nil {
		return fmt.Errorf("node update failed: %v", err)
	}
	if err := txn.Insert("index", &IndexEntry{"nodes", index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped error (%v) to identify the underlying store failure.
2. Restart the Nomad server agent to rebuild/reset the state store if corruption is suspected.
3. Restore the state store from a snapshot (nomad operator snapshot) if the backend is corrupted.
4. Retry the node event update once the store is healthy.

Example fix

// before
if err := store.UpsertNodeEvents(idx, nodeID, events); err != nil { return err }
// after
if err := store.UpsertNodeEvents(idx, nodeID, events); err != nil {
    logger.Error("node event upsert failed", "node", nodeID, "err", err)
    return err
}

node, err := store.NodeByID(nil, nodeID)
if err != nil {
    return fmt.Errorf("pre-check lookup failed: %w", err)
}

if err := store.UpsertNodeEvents(idx, nodeID, events); err != nil {
    if strings.HasPrefix(err.Error(), "node lookup failed") {
        logger.Error("state store lookup failed; check backing store health", "err", err)
    }
    return err
}

When it happens

Trigger: txn.First("nodes", "id", nodeID) returns a non-nil error — typically an underlying bolt/memdb transaction failure such as a corrupted store, txn misuse, or an I/O error on the backing store.

Common situations: State store backend I/O failure (disk full, corruption) during node event updates; internal bugs passing bad arguments to memdb; transaction already aborted when upsertNodeEvents is invoked within a composite write.

Related errors

• node update failed: %v
• failed to retrieve jobs for idempotency check
• failed to lookup state snapshot: %v
• volume update failed: %s: %v
• failed inserting variable: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/0c63aa2af58c53e2. Report an issue: GitHub.