hashicorp/nomad · error

host volume %s has nonexistent node ID %s

Error message

host volume %s has nonexistent node ID %s

What it means

UpsertHostVolume rejects a host volume whose NodeID does not correspond to an existing node in state store. Before persisting, it looks up the node via NodeByID; if the node row is missing (nil), the write is aborted so no orphaned volume referencing a ghost node is stored. This is an invariant check to keep host volumes anchored to a real, fingerprinted node.

Source

Thrown at nomad/state/state_store_host_volumes.go:91

		vol.CreateTime = old.CreateTime
	} else {
		vol.CreateIndex = index
	}

	err = s.enforceHostVolumeQuotaTxn(txn, index, vol, old, true)
	if err != nil {
		return err
	}

	// If the fingerprint is written from the node before the create RPC handler
	// completes, we'll never update from the initial pending, so reconcile that
	// here
	node, err := s.NodeByID(nil, vol.NodeID)
	if err != nil {
		return err
	}
	if node == nil {
		return fmt.Errorf("host volume %s has nonexistent node ID %s", vol.ID, vol.NodeID)
	}

	// prevent a race between node fingerprint and create RPC that could
	// switch a ready volume back to pending
	if _, ok := node.HostVolumes[vol.Name]; ok {
		vol.State = structs.HostVolumeStateReady
	}

	// Register RPCs for new volumes may not have the node pool set
	vol.NodePool = node.NodePool

	// Allocations are denormalized on read, so we don't want these to be
	// written to the state store.
	vol.Allocations = nil
	vol.ModifyIndex = index

	err = txn.Insert(TableHostVolumes, vol)
	if err != nil {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify the node exists with `nomad node status <node-id>` and use the correct, currently-registered node ID in the host volume spec.
2. Ensure the client node has registered and completed fingerprinting before creating host volumes on it.
3. If the node was replaced, re-register the host volume with the new node's ID.
4. If replaying state or snapshots, prune host volumes whose nodes no longer exist before upserting.

Example fix

// before
vol := &structs.HostVolume{ID: volID, NodeID: "old-node-id", Name: "data"}
stateStore.UpsertHostVolume(idx, vol)

// after
node, _ := stateStore.NodeByID(nil, nodeID)
if node != nil {
    vol := &structs.HostVolume{ID: volID, NodeID: node.ID, Name: "data"}
    stateStore.UpsertHostVolume(idx, vol)
}

node, err := s.NodeByID(nil, vol.NodeID)
if err != nil {
    return err
}
if node == nil {
    return fmt.Errorf("skip: node %s not registered", vol.NodeID)
}
// safe to upsert host volume now

func nodeExists(s *state.StateStore, nodeID string) bool {
    n, err := s.NodeByID(nil, nodeID)
    return err == nil && n != nil
}

err := s.UpsertHostVolume(idx, vol)
if err != nil && strings.Contains(err.Error(), "has nonexistent node ID") {
    // refresh node registration and retry once
}

When it happens

Trigger: Calling UpsertHostVolume (via the HostVolume Register RPC or state apply of a HostVolumeRegister message) with vol.NodeID set to a node ID that is not in the node table — e.g. a stale or fabricated ID, or registering the volume before the node has registered/fingerprinted.

Common situations: Operators hand-writing host volume registration against a node ID copied from a removed/replaced client; a race where a volume create RPC arrives before node registration completes; tooling that replays old state snapshots after nodes were garbage-collected.

Related errors

• ACL policy not found
• could not delete volume %s in use by alloc %s
• error parsing: root should be an object
• cannot specify Accessor ID
• network already configured but not found in state

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/463c5d663c4d7999. Report an issue: GitHub.