unslothai/unsloth · critical · TorchDeviceUnusableError

torch crashes when allocating on {device}; this install's to

Error message

torch crashes when allocating on {device}; this install's torch build does not match this machine

What it means

TorchDeviceUnusableError raised when torch crashes on allocation for BOTH the requested device (e.g. 'cuda') and CPU, checked via an actual allocation probe (device_can_allocate). The code first tries the requested device, falls back to CPU with a warning, and only raises when even CPU allocation crashes — meaning the installed torch binary is fundamentally incompatible with this machine (wrong CUDA/ROCm build, broken driver userspace, mismatched ABI).

Source

Thrown at studio/backend/core/rag/embeddings.py:114

    Fall back to CPU to preserve the embedding space. Raise only if CPU also
    crashes, allowing the caller to select the GGUF backend."""
    device = _device()
    if device == "cpu":
        return device

    from utils.torch_device_probe import device_can_allocate

    if device_can_allocate(device):
        return device
    if device_can_allocate("cpu"):
        logger.warning(
            "torch cannot allocate on %s without crashing; loading the embedding model "
            "on CPU instead. This install's torch build does not match this machine.",
            device,
        )
        return "cpu"
    raise TorchDeviceUnusableError(
        f"torch crashes when allocating on {device}; this install's torch build does "
        "not match this machine"
    )


_torchao_stub_done = False


def _install_torchao_stub_once() -> None:
    """Neutralize torchao before importing sentence-transformers. On Windows ROCm,
    torchao (pulled in by transformers.quantizers) imports an absent c10d backend
    and aborts, dropping the embedder to llama-server. Workers stub it too; the
    embedder runs in the main process. No-op elsewhere; runs once under ``_lock``."""
    global _torchao_stub_done
    if _torchao_stub_done:
        return
    _torchao_stub_done = True
    from core._torchao_stub import install_torchao_windows_rocm_stub

View on GitHub (pinned to 203007d190)

Solutions

1. Reinstall torch matching this machine: choose the wheel for the actual driver/CUDA (or CPU-only wheel) from pytorch.org.
2. Update the GPU driver to the version the installed torch build requires.
3. As an immediate unblock, set RAG_EMBED_BACKEND=llama-server — it avoids torch entirely for embeddings.
4. In containers, ensure the NVIDIA container toolkit mounts libcuda for the torch CUDA build.
5. Run python -c "import torch; torch.zeros(1)" to confirm the fix before restarting the app.

Example fix

# before (wrong wheel for driver)
pip install torch  # defaults to a CUDA build the driver cannot serve

# after
pip install torch --index-url https://download.pytorch.org/whl/cpu  # or the matching cuXXX index

from utils.torch_device_probe import device_can_allocate

def pick_device(requested: str) -> str | None:
    if device_can_allocate(requested):
        return requested
    if device_can_allocate("cpu"):
        return "cpu"
    return None  # torch unusable -> choose non-torch backend before importing it

try:
    device = _safe_device(config.EMBED_DEVICE)
except TorchDeviceUnusableError:
    # torch fundamentally broken on this machine; avoid it entirely
    os.environ["RAG_EMBED_BACKEND"] = "llama-server"
    device = None

When it happens

Trigger: Loading the sentence-transformers embedding backend on a machine where the pip-installed torch wheel's CUDA version has no compatible driver (e.g. cu124 wheel on cu117 driver) so any tensor allocation segfaults; a ROCm build on non-AMD hardware; a glibc/ABI mismatch that crashes even CPU allocation.

Common situations: Copying a venv or requirements lockfile between machines with different GPUs/drivers; installing torch from a PyTorch index that defaults to the wrong platform wheel; Windows systems with outdated NVIDIA drivers; container images where libcuda is absent but the CUDA wheel expects it.

Related errors

• GPU selection is unavailable on this host: {exc}
• Requested GPU {wanted} but none of them are visible to this
• The current inference worker did not exit and still holds GP
• Invalid gpu_ids {requested_ids}: explicit physical GPU IDs a
• Invalid gpu_ids {requested_ids}: duplicate GPU IDs are not a

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/f22b4fdefc6b6416. Report an issue: GitHub.