hashicorp/nomad · error

node not found: %s

Error message

node not found: %s

What it means

deleteNodeTxn fails when a node ID in the requested batch does not exist in the 'nodes' table (txn.First returns nil). Deletion is aborted atomically so previously processed node deletions in the same txn are rolled back. Raised through StateStore.DeleteNode.

Source

Thrown at nomad/state/state_store.go:1080

	err := s.deleteNodeTxn(txn, index, nodes)
	if err != nil {
		return nil
	}
	return txn.Commit()
}

func (s *StateStore) deleteNodeTxn(txn *txn, index uint64, nodes []string) error {
	if len(nodes) == 0 {
		return fmt.Errorf("node ids missing")
	}

	for _, nodeID := range nodes {
		existing, err := txn.First("nodes", "id", nodeID)
		if err != nil {
			return fmt.Errorf("node lookup failed: %s: %v", nodeID, err)
		}
		if existing == nil {
			return fmt.Errorf("node not found: %s", nodeID)
		}

		// Delete the node
		if err := txn.Delete("nodes", existing); err != nil {
			return fmt.Errorf("node delete failed: %s: %v", nodeID, err)
		}

		node := existing.(*structs.Node)
		if err := deleteNodeCSIPlugins(txn, node, index); err != nil {
			return fmt.Errorf("csi plugin delete failed: %v", err)
		}
		if node.GCVolumesOnNodeGC {
			if err := s.deleteHostVolumesOnNode(txn, index, node.ID); err != nil {
				return fmt.Errorf("dynamic host volume delete failed: %v", err)
			}
		}
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify the node ID exists via the node lookup API before calling DeleteNode.
2. Treat 'node not found' as an idempotent success in callers and skip it.
3. Filter the requested ID list to nodes that still exist to avoid aborting the whole batch.
4. Check for concurrent deregistration flows that may have already removed the node.

Example fix

// before
if err := store.DeleteNode(idx, nodeIDs); err != nil { return err }
// after
for _, id := range nodeIDs {
    if _, err := store.NodeByID(nil, id); err != nil || exists == nil {
        continue // skip already-deleted nodes
    }
}
err := store.DeleteNode(idx, existingIDs)

for _, id := range nodeIDs {
    n, err := store.NodeByID(nil, id)
    if err != nil {
        return err
    }
    if n == nil {
        return fmt.Errorf("node %s does not exist", id)
    }
}

func nodeExists(s *state.StateStore, id string) bool {
    n, err := s.NodeByID(nil, id)
    return err == nil && n != nil
}

if err := store.DeleteNode(idx, nodeIDs); err != nil {
    if strings.HasPrefix(err.Error(), "node not found:") {
        // idempotent success or skip missing node
        return nil
    }
    return err
}

When it happens

Trigger: Calling DeleteNode with a node ID that was already deregistered, never registered, or mistyped; also occurs when two concurrent deregistrations race and the second one targets the now-deleted node.

Common situations: Retry/duplicate delivery of a deregistration request; operators deleting nodes via API with stale IDs after garbage collection removed them; typo'd node UUID in a script.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• node ids missing
• node not found
• error parsing: root should be an object
• cannot specify Accessor ID
• network already configured but not found in state

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/e6db9e90ea5fb38c. Report an issue: GitHub.