unslothai/unsloth · error · ValueError

Requested GPU {wanted} but none of them are visible to this

Error message

Requested GPU {wanted} but none of them are visible to this process (visible: {visible}). Clear the GPU selection to use the default device.

What it means

The requested physical GPU ids, after filtering through resolve_requested_gpu_ids and the parent's CUDA_VISIBLE_DEVICES mask, have no intersection with the ids actually visible to this process. The load is refused with the visible list rather than quietly running on a device the user did not choose.

Source

Thrown at studio/backend/core/inference/diffusion_device.py:196

    """
    wanted = sorted({int(gpu_id) for gpu_id in gpu_ids or ()})
    if not wanted:
        return None
    try:
        from utils.hardware.hardware import (
            get_parent_visible_gpu_ids,
            resolve_requested_gpu_ids,
        )
    except Exception as exc:  # noqa: BLE001 -- without the hardware layer the mask is unknowable
        raise ValueError(f"GPU selection is unavailable on this host: {exc}") from exc
    allowed = resolve_requested_gpu_ids(wanted)
    visible = get_parent_visible_gpu_ids()
    # Torch enumerates the parent-visible list in order, so its ordinal for a physical id is that
    # id's position in the mask. Unmasked, the layer reports range(physical count) and this is
    # the identity mapping.
    ordinals = [visible.index(gpu_id) for gpu_id in allowed if gpu_id in visible]
    if not ordinals:
        raise ValueError(
            f"Requested GPU {wanted} but none of them are visible to this process "
            f"(visible: {visible}). Clear the GPU selection to use the default device."
        )
    if len(ordinals) == 1:
        return ordinals[0]
    if not allow_ranking:
        return None

    def _free_vram(ordinal: int) -> int:
        try:
            import torch
            return int(torch.cuda.mem_get_info(ordinal)[0])
        except Exception:  # noqa: BLE001 -- an unreadable card sorts last rather than failing the load
            return -1

    return max(ordinals, key = lambda ordinal: (_free_vram(ordinal), -ordinal))

View on GitHub (pinned to 203007d190)

Solutions

Pick one of the ids listed in the error's '(visible: ...)' set
Widen or clear CUDA_VISIBLE_DEVICES for the backend process so the requested card is visible, then restart it
Clear the GPU selection in the request (no gpu_ids) to fall back to the default device

Example fix

# before
# CUDA_VISIBLE_DEVICES=4,5, request:
ordinal = resolve_gpu_ordinal(gpu_ids=[0])  # ValueError

# after
ordinal = resolve_gpu_ordinal(gpu_ids=[4])   # a visible physical id
# or unset the mask / clear the selection:
ordinal = resolve_gpu_ordinal(gpu_ids=None)

Defensive patterns

Strategy: validation

Validate before calling

from utils.hardware.hardware import get_parent_visible_gpu_ids

def gpus_requestable(gpu_ids) -> bool:
    if not gpu_ids:
        return True
    visible = get_parent_visible_gpu_ids()
    return bool(set(map(int, gpu_ids)) & set(visible))

Try / catch

try:
    ordinal = resolve_gpu_ordinal(gpu_ids)
except ValueError as e:
    # message includes the visible list; offer those or the default device
    return bad_request(str(e))

Prevention

Read CUDA_VISIBLE_DEVICES before offering GPU choices in the UI; show only parent-visible ids
Re-read the mask at request time — orchestrators can change it between inventory and use
Remember ids are PHYSICAL ids; torch ordinals are positions in the visible list, never send ordinals

When it happens

Trigger: Calling the GPU-selection resolve with gpu_ids like [0,1] while CUDA_VISIBLE_DEVICES=4,5 (or unset-but-requested cards beyond the mask); requesting a card the container/host mask excludes; nvidia-smi reports the card but the process mask does not include it.

Common situations: Service launched under a container orchestrator or systemd unit that sets CUDA_VISIBLE_DEVICES; user selects GPU 0 in the UI not realizing the process only sees physical 4,5; mask changed after the UI read the host inventory.

Related errors

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/12e6ea462b0337ec. Report an issue: GitHub.