unslothai/unsloth · error · ValueError

{context} does not support models loaded with CPU or disk of

Error message

{context} does not support models loaded with CPU or disk offload. device_map='{device_map}' produced offloaded modules: {example}

What it means

raise_if_offloaded inspects a loaded model's device_map placements and raises when any modules ended up on CPU or disk (via get_offloaded_device_map_entries). The message names the calling context, the device_map string used, and up to 5 example offloaded modules with placements. It exists because many operations (e.g. LoRA/quantization paths) are incompatible with accelerate offloading.

Source

Thrown at studio/backend/utils/hardware/hardware.py:3888

        return {}
    return {
        module_name: placement
        for module_name, placement in hf_device_map.items()
        if placement in ("cpu", "disk")
    }


def raise_if_offloaded(
    model,
    device_map: str,
    context: str = "Loading",
) -> None:
    """Raise ``ValueError`` if *model* has modules offloaded to CPU or disk."""
    offloaded = get_offloaded_device_map_entries(model)
    if not offloaded:
        return
    example = ", ".join(f"{name}={placement}" for name, placement in list(offloaded.items())[:5])
    raise ValueError(
        f"{context} does not support models loaded with CPU or disk offload. "
        f"device_map='{device_map}' produced offloaded modules: {example}"
    )


def get_torch_device_str() -> str:
    """
    Return the torch device string for the detected hardware.
    E.g. "cuda", "xpu", or "cpu".
    """
    device = get_device()
    if device == DeviceType.CUDA:
        return "cuda"
    elif device == DeviceType.XPU:
        return "xpu"
    return "cpu"

View on GitHub (pinned to 203007d190)

Solutions

Free VRAM (close other jobs) or restart with a clean GPU so the model fits entirely.
Use a smaller model or quantized variant (4-bit/8-bit) so all modules fit on GPU.
Load with an explicit GPU-only device_map (e.g. device_map={'': 0}) so failure to fit surfaces at load time instead.
Add GPUs to the node / increase gpu_ids so sharding keeps everything on devices.
If CPU offload is genuinely intended, use a code path that supports it instead of the context that raises.

Example fix

# before
model = load(model_name, device_map="auto")  # partially offloaded to CPU
raise_if_offloaded(model, "auto", "Training")  # ValueError

# after
model = load(quantize_4bit(model_name), device_map={"": 0})  # fits fully on GPU

Defensive patterns

Strategy: validation

Validate before calling

def model_fits_without_offload(model) -> bool:
    from utils.hardware.hardware import get_offloaded_device_map_entries
    return not get_offloaded_device_map_entries(model)

# after load: assert model_fits_without_offload(model) before training

Try / catch

try:
    raise_if_offloaded(model, device_map="auto", context="Training")
except ValueError as e:
    if "offloaded modules" in str(e):
        del model
        model = load_quantized(model_name, bits=4, device_map={"": 0})
        raise_if_offloaded(model, "balanced", "Training")
    else:
        raise

Prevention

Free VRAM before loading (check nvidia-smi for stray processes).
Use quantized (4/8-bit) variants for large models on small GPUs.
Load with device_map={'': 0} so misfit fails fast at load, not mid-pipeline.
Add GPUs or reduce concurrent jobs when sharding would spill to CPU.

When it happens

Trigger: Loading a model with device_map='auto' (or 'balanced') when it does not fit in available VRAM, causing accelerate to place layers on CPU/disk, then calling an operation that invokes raise_if_offloaded.

Common situations: GPU too small for the chosen model (7B fp16 needs ~14GB+); other processes consuming VRAM; device_map='auto' on a multi-GPU node sharding to CPU; someone requesting device_map='auto' expecting pure-GPU placement.

Related errors

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/e607a275094db1b7. Report an issue: GitHub.