{"record":{"id":"f22b4fdefc6b6416","repo":"unslothai/unsloth","slug":"torch-crashes-when-allocating-on-device-this-in","errorCode":null,"errorMessage":"torch crashes when allocating on {device}; this install's torch build does not match this machine","messagePattern":"torch crashes when allocating on (.+?); this install's torch build does not match this machine","errorType":"exception","errorClass":"TorchDeviceUnusableError","httpStatus":null,"severity":"critical","filePath":"studio/backend/core/rag/embeddings.py","lineNumber":114,"sourceCode":"\n    Fall back to CPU to preserve the embedding space. Raise only if CPU also\n    crashes, allowing the caller to select the GGUF backend.\"\"\"\n    device = _device()\n    if device == \"cpu\":\n        return device\n\n    from utils.torch_device_probe import device_can_allocate\n\n    if device_can_allocate(device):\n        return device\n    if device_can_allocate(\"cpu\"):\n        logger.warning(\n            \"torch cannot allocate on %s without crashing; loading the embedding model \"\n            \"on CPU instead. This install's torch build does not match this machine.\",\n            device,\n        )\n        return \"cpu\"\n    raise TorchDeviceUnusableError(\n        f\"torch crashes when allocating on {device}; this install's torch build does \"\n        \"not match this machine\"\n    )\n\n\n_torchao_stub_done = False\n\n\ndef _install_torchao_stub_once() -> None:\n    \"\"\"Neutralize torchao before importing sentence-transformers. On Windows ROCm,\n    torchao (pulled in by transformers.quantizers) imports an absent c10d backend\n    and aborts, dropping the embedder to llama-server. Workers stub it too; the\n    embedder runs in the main process. No-op elsewhere; runs once under ``_lock``.\"\"\"\n    global _torchao_stub_done\n    if _torchao_stub_done:\n        return\n    _torchao_stub_done = True\n    from core._torchao_stub import install_torchao_windows_rocm_stub","sourceCodeStart":96,"sourceCodeEnd":132,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/core/rag/embeddings.py#L96-L132","documentation":"TorchDeviceUnusableError raised when torch crashes on allocation for BOTH the requested device (e.g. 'cuda') and CPU, checked via an actual allocation probe (device_can_allocate). The code first tries the requested device, falls back to CPU with a warning, and only raises when even CPU allocation crashes — meaning the installed torch binary is fundamentally incompatible with this machine (wrong CUDA/ROCm build, broken driver userspace, mismatched ABI).","triggerScenarios":"Loading the sentence-transformers embedding backend on a machine where the pip-installed torch wheel's CUDA version has no compatible driver (e.g. cu124 wheel on cu117 driver) so any tensor allocation segfaults; a ROCm build on non-AMD hardware; a glibc/ABI mismatch that crashes even CPU allocation.","commonSituations":"Copying a venv or requirements lockfile between machines with different GPUs/drivers; installing torch from a PyTorch index that defaults to the wrong platform wheel; Windows systems with outdated NVIDIA drivers; container images where libcuda is absent but the CUDA wheel expects it.","solutions":["Reinstall torch matching this machine: choose the wheel for the actual driver/CUDA (or CPU-only wheel) from pytorch.org.","Update the GPU driver to the version the installed torch build requires.","As an immediate unblock, set RAG_EMBED_BACKEND=llama-server — it avoids torch entirely for embeddings.","In containers, ensure the NVIDIA container toolkit mounts libcuda for the torch CUDA build.","Run python -c \"import torch; torch.zeros(1)\" to confirm the fix before restarting the app."],"exampleFix":"# before (wrong wheel for driver)\npip install torch  # defaults to a CUDA build the driver cannot serve\n\n# after\npip install torch --index-url https://download.pytorch.org/whl/cpu  # or the matching cuXXX index","handlingStrategy":"fallback","validationCode":"from utils.torch_device_probe import device_can_allocate\n\ndef pick_device(requested: str) -> str | None:\n    if device_can_allocate(requested):\n        return requested\n    if device_can_allocate(\"cpu\"):\n        return \"cpu\"\n    return None  # torch unusable -> choose non-torch backend before importing it","typeGuard":null,"tryCatchPattern":"try:\n    device = _safe_device(config.EMBED_DEVICE)\nexcept TorchDeviceUnusableError:\n    # torch fundamentally broken on this machine; avoid it entirely\n    os.environ[\"RAG_EMBED_BACKEND\"] = \"llama-server\"\n    device = None","preventionTips":["Pin torch wheels to the index matching your driver (cuXXX or cpu) in requirements.","Add a startup probe (one small allocation) so a broken torch fails fast at boot, not mid-ingest.","Keep RAG_EMBED_BACKEND=llama-server as the documented escape hatch for broken torch installs."],"tags":["torch","cuda","environment","gpu","embeddings"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}