{"record":{"id":"463c5d663c4d7999","repo":"hashicorp/nomad","slug":"host-volume-s-has-nonexistent-node-id-s","errorCode":null,"errorMessage":"host volume %s has nonexistent node ID %s","messagePattern":"host volume (.+?) has nonexistent node ID (.+?)","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store_host_volumes.go","lineNumber":91,"sourceCode":"\t\tvol.CreateTime = old.CreateTime\n\t} else {\n\t\tvol.CreateIndex = index\n\t}\n\n\terr = s.enforceHostVolumeQuotaTxn(txn, index, vol, old, true)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\t// If the fingerprint is written from the node before the create RPC handler\n\t// completes, we'll never update from the initial pending, so reconcile that\n\t// here\n\tnode, err := s.NodeByID(nil, vol.NodeID)\n\tif err != nil {\n\t\treturn err\n\t}\n\tif node == nil {\n\t\treturn fmt.Errorf(\"host volume %s has nonexistent node ID %s\", vol.ID, vol.NodeID)\n\t}\n\n\t// prevent a race between node fingerprint and create RPC that could\n\t// switch a ready volume back to pending\n\tif _, ok := node.HostVolumes[vol.Name]; ok {\n\t\tvol.State = structs.HostVolumeStateReady\n\t}\n\n\t// Register RPCs for new volumes may not have the node pool set\n\tvol.NodePool = node.NodePool\n\n\t// Allocations are denormalized on read, so we don't want these to be\n\t// written to the state store.\n\tvol.Allocations = nil\n\tvol.ModifyIndex = index\n\n\terr = txn.Insert(TableHostVolumes, vol)\n\tif err != nil {","sourceCodeStart":73,"sourceCodeEnd":109,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store_host_volumes.go#L73-L109","documentation":"UpsertHostVolume rejects a host volume whose NodeID does not correspond to an existing node in state store. Before persisting, it looks up the node via NodeByID; if the node row is missing (nil), the write is aborted so no orphaned volume referencing a ghost node is stored. This is an invariant check to keep host volumes anchored to a real, fingerprinted node.","triggerScenarios":"Calling UpsertHostVolume (via the HostVolume Register RPC or state apply of a HostVolumeRegister message) with vol.NodeID set to a node ID that is not in the node table — e.g. a stale or fabricated ID, or registering the volume before the node has registered/fingerprinted.","commonSituations":"Operators hand-writing host volume registration against a node ID copied from a removed/replaced client; a race where a volume create RPC arrives before node registration completes; tooling that replays old state snapshots after nodes were garbage-collected.","solutions":["Verify the node exists with `nomad node status <node-id>` and use the correct, currently-registered node ID in the host volume spec.","Ensure the client node has registered and completed fingerprinting before creating host volumes on it.","If the node was replaced, re-register the host volume with the new node's ID.","If replaying state or snapshots, prune host volumes whose nodes no longer exist before upserting."],"exampleFix":"// before\nvol := &structs.HostVolume{ID: volID, NodeID: \"old-node-id\", Name: \"data\"}\nstateStore.UpsertHostVolume(idx, vol)\n\n// after\nnode, _ := stateStore.NodeByID(nil, nodeID)\nif node != nil {\n    vol := &structs.HostVolume{ID: volID, NodeID: node.ID, Name: \"data\"}\n    stateStore.UpsertHostVolume(idx, vol)\n}","handlingStrategy":"validation","validationCode":"node, err := s.NodeByID(nil, vol.NodeID)\nif err != nil {\n    return err\n}\nif node == nil {\n    return fmt.Errorf(\"skip: node %s not registered\", vol.NodeID)\n}\n// safe to upsert host volume now","typeGuard":"func nodeExists(s *state.StateStore, nodeID string) bool {\n    n, err := s.NodeByID(nil, nodeID)\n    return err == nil && n != nil\n}","tryCatchPattern":"err := s.UpsertHostVolume(idx, vol)\nif err != nil && strings.Contains(err.Error(), \"has nonexistent node ID\") {\n    // refresh node registration and retry once\n}","preventionTips":["Always resolve node IDs from `nomad node status` or NodeByID, never from cached/stale inventories.","Order operations: register/fingerprint the node before creating host volumes on it.","In automation, validate node existence before every host volume upsert.","Watch for node garbage collection between validation and upsert."],"tags":["nomad","state-store","host-volume","referential-integrity"],"backgroundTag":"referenced-entity-not-found","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}