hashicorp/nomad · error · ErrUnknownNode

%w %s

Error message

%w %s

What it means

controllerPublishVolume wraps structs.ErrUnknownNode when the allocation's node (alloc.NodeID) cannot be found in the Nomad state store. The error carries the node ID so operators can identify which client is missing. Nomad must resolve the Nomad node before it can map it to the storage provider's external node ID for the CSI ControllerAttachVolume RPC.

Source

Thrown at nomad/csi_endpoint.go:561

	}
	if alloc == nil {
		return fmt.Errorf("%s: %s", structs.ErrUnknownAllocationPrefix, req.AllocationID)
	}

	// Some plugins support controllers for create/snapshot but not attach. So
	// if there's no plugin or the plugin doesn't attach volumes, then we can
	// skip the controller publish workflow and return nil.
	if plug == nil || !plug.HasControllerCapability(structs.CSIControllerSupportsAttachDetach) {
		return nil
	}

	// get Nomad's ID for the client node (not the storage provider's ID)
	targetNode, err := state.NodeByID(ws, alloc.NodeID)
	if err != nil {
		return err
	}
	if targetNode == nil {
		return fmt.Errorf("%w %s", structs.ErrUnknownNode, alloc.NodeID)
	}

	// if the RPC is sent by a client node, it may not know the claim's
	// external node ID.
	if req.ExternalNodeID == "" {
		externalNodeID, err := v.lookupExternalNodeID(vol, req.ToClaim())
		if err != nil {
			return fmt.Errorf("missing external node ID: %v", err)
		}
		req.ExternalNodeID = externalNodeID
	}

	method := "ClientCSI.ControllerAttachVolume"
	cReq := &cstructs.ClientCSIControllerAttachVolumeRequest{
		VolumeID:        vol.RemoteID(),
		ClientCSINodeID: req.ExternalNodeID,
		AttachmentMode:  req.AttachmentMode,
		AccessMode:      req.AccessMode,

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check `nomad node status <node-id>`; if the node is down, bring the Nomad client back online so it re-registers (node IDs are stable via the client's data dir).
2. If the node is permanently gone, stop/purge the allocations on it (`nomad node drain` then `nomad job stop`) so stale claims are released.
3. Reschedule the workload onto a live node to generate a fresh claim referencing a registered node.
4. If the node was erroneously GC'd, check server gc thresholds (node_gc_threshold) and client heartbeat TTL settings.

Example fix

// before: retrying claim against a deregistered node's alloc
client.CSIVolumes().Claim(volID, namespace, writeOpts, claimReq)
// after: verify the node is registered first
node, _, _ := client.Nodes().Info(alloc.NodeID, nil)
if node != nil && node.Status == "ready" {
    client.CSIVolumes().Claim(volID, namespace, writeOpts, claimReq)
}

node, _, err := client.Nodes().Info(alloc.NodeID, nil)
if err != nil || node == nil {
    return fmt.Errorf("node %s is not registered; node may have been GC'd", alloc.NodeID)
}
if node.Status != "ready" {
    return fmt.Errorf("node %s is %s; not eligible for volume claims", node.ID, node.Status)
}

if err != nil && strings.Contains(err.Error(), structs.ErrUnknownNode.Error()) {
    // node deregistered: drain/re-reschedule the workload
}

When it happens

Trigger: Raised in controllerPublishVolume (via CSIVolume.Claim) when state.NodeByID(ws, alloc.NodeID) returns nil: the allocation exists but the node it was placed on has been deregistered/garbage-collected from the node registry. Typically the node failed its heartbeat and was GC'd while an alloc on it still claims a volume.

Common situations: A client node lost connectivity long enough for its heartbeat to expire and the server garbage-collected it; its allocations remain and a volume claim retry hits the missing node. Also occurs in disconnected-client scenarios, after manual `nomad node eligibility`/purge operations, or when state was partially restored.

Related errors

• %w: %v
• plugin instance %q is not healthy
• csi_plugins delete error: %v
• volume lookup failed: %s %v
• error parsing: root should be an object

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/b85399e7b05d7b12. Report an issue: GitHub.