hashicorp/nomad · error
node not found
Error message
node not found
What it means
updateNodeStatusTxn returns 'node not found' when UpdateNodeStatus is called for a node ID with no row in the 'nodes' table. The status update is rejected because the store requires the existing node record to copy and mutate. Surfaced through StateStore.UpdateNodeStatus.
Solutions
- Confirm the node is registered (NodeByID) before updating its status.
- Ignore/handle this as a benign race when the heartbeat races deregistration.
- Re-register the node (UpsertNode) if it should exist, then retry the status update.
- Validate the node UUID for typos in scripts or configs.
Example fix
// before
err := store.UpdateNodeStatus(idx, nodeID, "ready", now)
// after
node, _ := store.NodeByID(nil, nodeID)
if node != nil {
err := store.UpdateNodeStatus(idx, nodeID, "ready", now)
} Defensive patterns
Strategy: try-catch
Validate before calling
node, err := store.NodeByID(nil, req.NodeID)
if err != nil {
return err
}
if node == nil {
return fmt.Errorf("cannot update status: node %s not registered", req.NodeID)
} Type guard
func nodeExists(s *state.StateStore, id string) bool {
n, err := s.NodeByID(nil, id)
return err == nil && n != nil
} Try / catch
if err := store.UpdateNodeStatus(idx, nodeID, status, now); err != nil {
if err.Error() == "node not found" {
logger.Warn("status update skipped; node gone", "node", nodeID)
return nil
}
return err
} Prevention
- Ignore heartbeat failures for nodes that were concurrently deregistered.
- Re-register the node if it should exist before updating status.
- Validate node UUIDs in tooling to catch typos.
- Log and monitor not-found heartbeats as potential GC/registration races.
When it happens
Trigger: Calling UpdateNodeStatus with req.NodeID that is unregistered or already deregistered/garbage-collected; heartbeats arriving from a node whose registration was removed; a stale node ID in a restore snapshot.
Common situations: Client heartbeats racing with operator deregistration; node was purged between event emit and status write; wrong/typo'd node UUID in a tooling script.
Understand the failure class
Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.
Related errors
- node not found
- ACL auth method lookup failed
- ACL auth method lookup failed
- ACL binding rule deletion failed
- ACL binding rule insert failed
AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04).
Data as JSON: /api/errors/a59d0244bb922a11.
Report an issue: GitHub.
Appendix: source
Thrown at nomad/state/state_store.go:1131
txn := s.db.WriteTxnMsgT(msgType, index)
defer txn.Abort()
if err := s.updateNodeStatusTxn(txn, req); err != nil {
return err
}
return txn.Commit()
}
func (s *StateStore) updateNodeStatusTxn(txn *txn, req *structs.NodeUpdateStatusRequest) error {
// Lookup the node
existing, err := txn.First(TableNodes, indexID, req.NodeID)
if err != nil {
return fmt.Errorf("node lookup failed: %v", err)
}
if existing == nil {
return fmt.Errorf("node not found")
}
// Copy the existing node
existingNode := existing.(*structs.Node)
copyNode := existingNode.Copy()
copyNode.StatusUpdatedAt = req.UpdatedAt
// If the request has a signing key ID, we should update the node reference
// to this. We need to check for the empty string, as a new identity won't
// always be generated, and we don't want to overwrite the exiting entry
// with an empty string.
if req.IdentitySigningKeyID != "" {
copyNode.IdentitySigningKeyID = req.IdentitySigningKeyID
}
// Add the event if given
if req.NodeEvent != nil {
appendNodeEvents(txn.Index, copyNode, []*structs.NodeEvent{req.NodeEvent})View on GitHub (pinned to 482b49bf1a)