hashicorp/nomad · error

volume cannot be removed from unknown node without force=tru

Error message

volume cannot be removed from unknown node without force=true

What it means

This error comes from Nomad's host-volume DELETE endpoint (nomad/host_volume_endpoint.go:712). Deleting a host volume requires contacting the node that hosts it so the client can release/deregister the volume; when the node is unknown or unreachable (IsErrUnknownNode / IsErrNoNodeConn), the server refuses to delete unless args.Force is set, to avoid silently removing a volume that may still exist on the node.

Source

Thrown at nomad/host_volume_endpoint.go:712

	vol, err := snap.HostVolumeByID(nil, ns, id, true)
	if err != nil {
		return fmt.Errorf("could not query host volume: %w", err)
	}
	if vol == nil {
		return fmt.Errorf("no such volume: %s", id)
	}
	if len(vol.Allocations) > 0 {
		allocIDs := helper.ConvertSlice(vol.Allocations,
			func(a *structs.AllocListStub) string { return a.ID })
		return fmt.Errorf("volume %s in use by allocations: %v", id, allocIDs)
	}

	// serialize client RPC and raft write per volume ID
	index, err := v.serializeCall(vol.ID, "delete", func() (uint64, error) {
		if err := v.deleteVolume(vol); err != nil {
			if structs.IsErrUnknownNode(err) || structs.IsErrNoNodeConn(err) {
				if !args.Force {
					return 0, fmt.Errorf(
						"volume cannot be removed from unknown node without force=true")
				}
			} else {
				return 0, err
			}
		}
		_, idx, err := v.srv.raftApply(structs.HostVolumeDeleteRequestType, args)
		if err != nil {
			v.logger.Error("raft apply failed", "error", err, "method", "delete")
			return 0, err
		}
		return idx, nil
	})
	if err != nil {
		return err
	}

	reply.Index = index

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Pass Force=true in the delete request (or run `nomad volume delete -force <volume_id>`) to remove the volume registration without contacting the node.
2. Re-register/restart the node (start the Nomad agent) so the server can connect and perform a graceful delete, then delete without force.
3. Deregister the dead node properly (`nomad node status` to find it, then purge/let it reap) if it should no longer exist, and clean up the volume afterward.
4. Verify the volume's NodeID (`nomad volume status <id>`) — if it points at a node that no longer exists, force delete is the correct operation.

Example fix

// before
curl -X DELETE http://localhost:4646/v1/volumes/my-volume?namespace=default

// after
curl -X DELETE 'http://localhost:4646/v1/volumes/my-volume?namespace=default&force=true'
// or CLI: nomad volume delete -force my-volume

vol, _, err := client.HostVolumes().Get(ctx, "my-volume", nil)
if err != nil { /* volume unknown; nothing to delete */ }
node, _, err := client.Nodes().Info(vol.NodeID, nil)
if err == nil && node.Status != "down" {
    // safe graceful delete
    _, err = client.HostVolumes().Delete(vol.ID, vol.Namespace, false, nil)
} else {
    // node unknown/down: force required
    _, err = client.HostVolumes().Delete(vol.ID, vol.Namespace, true, nil)
}

func nodeReachable(n *api.Node) bool {
	return n != nil && n.Status != "down" && n.Status != "disconnected"
}

_, err := client.HostVolumes().Delete(vol.ID, vol.Namespace, false, nil)
if err != nil && strings.Contains(err.Error(), "unknown node without force=true") {
	_, err = client.HostVolumes().Delete(vol.ID, vol.Namespace, true, nil)
}

When it happens

Trigger: Calling the HostVolume.Delete RPC (or `nomad volume delete <id>`) for a volume whose registered node has been deregistered, is down, or has no established connection, without setting Force=true in the delete request.

Common situations: A node was drained, decommissioned, or its agent crashed and was removed from the cluster, leaving a stale host volume registration; operators then try to clean up the volume via API/CLI and get blocked. Also common after node ID changes or restoring state snapshots.

Related errors

• CSI.ControllerAttachVolume: VolumeID is required
• CSI.ControllerAttachVolume: ClientCSINodeID is required
• CSI.ControllerDetachVolume: VolumeID is required
• CSI.ControllerDetachVolume: ClientCSINodeID is required
• CSI.NodeDetachVolume: PluginID is required

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/00cb4811eec1e774. Report an issue: GitHub.