hashicorp/nomad · error

node not found

Error message

node not found

What it means

updateNodeStatusTxn returns 'node not found' when UpdateNodeStatus is called for a node ID with no row in the 'nodes' table. The status update is rejected because the store requires the existing node record to copy and mutate. Surfaced through StateStore.UpdateNodeStatus.

Source

Thrown at nomad/state/state_store.go:1131

	txn := s.db.WriteTxnMsgT(msgType, index)
	defer txn.Abort()

	if err := s.updateNodeStatusTxn(txn, req); err != nil {
		return err
	}

	return txn.Commit()
}

func (s *StateStore) updateNodeStatusTxn(txn *txn, req *structs.NodeUpdateStatusRequest) error {

	// Lookup the node
	existing, err := txn.First(TableNodes, indexID, req.NodeID)
	if err != nil {
		return fmt.Errorf("node lookup failed: %v", err)
	}
	if existing == nil {
		return fmt.Errorf("node not found")
	}

	// Copy the existing node
	existingNode := existing.(*structs.Node)
	copyNode := existingNode.Copy()
	copyNode.StatusUpdatedAt = req.UpdatedAt

	// If the request has a signing key ID, we should update the node reference
	// to this. We need to check for the empty string, as a new identity won't
	// always be generated, and we don't want to overwrite the exiting entry
	// with an empty string.
	if req.IdentitySigningKeyID != "" {
		copyNode.IdentitySigningKeyID = req.IdentitySigningKeyID
	}

	// Add the event if given
	if req.NodeEvent != nil {
		appendNodeEvents(txn.Index, copyNode, []*structs.NodeEvent{req.NodeEvent})

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Confirm the node is registered (NodeByID) before updating its status.
2. Ignore/handle this as a benign race when the heartbeat races deregistration.
3. Re-register the node (UpsertNode) if it should exist, then retry the status update.
4. Validate the node UUID for typos in scripts or configs.

Example fix

// before
err := store.UpdateNodeStatus(idx, nodeID, "ready", now)
// after
node, _ := store.NodeByID(nil, nodeID)
if node != nil {
    err := store.UpdateNodeStatus(idx, nodeID, "ready", now)
}

node, err := store.NodeByID(nil, req.NodeID)
if err != nil {
    return err
}
if node == nil {
    return fmt.Errorf("cannot update status: node %s not registered", req.NodeID)
}

func nodeExists(s *state.StateStore, id string) bool {
    n, err := s.NodeByID(nil, id)
    return err == nil && n != nil
}

if err := store.UpdateNodeStatus(idx, nodeID, status, now); err != nil {
    if err.Error() == "node not found" {
        logger.Warn("status update skipped; node gone", "node", nodeID)
        return nil
    }
    return err
}

When it happens

Trigger: Calling UpdateNodeStatus with req.NodeID that is unregistered or already deregistered/garbage-collected; heartbeats arriving from a node whose registration was removed; a stale node ID in a restore snapshot.

Common situations: Client heartbeats racing with operator deregistration; node was purged between event emit and status write; wrong/typo'd node UUID in a tooling script.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• node not found: %s
• error parsing: root should be an object
• cannot specify Accessor ID
• network already configured but not found in state
• eval not found

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/a59d0244bb922a11. Report an issue: GitHub.