unslothai/unsloth · error · ValueError

Invalid gpu_ids {requested_ids}: requested GPUs {disallowed_

Error message

Invalid gpu_ids {requested_ids}: requested GPUs {disallowed_ids} are outside the parent-visible set {parent_visible_ids}

What it means

The authoritative final GPU check: every requested id must be a member of the parent-visible set (devices the parent process can see after applying CUDA_VISIBLE_DEVICES/ZE_AFFINITY_MASK). It fires for ids that are valid physical ids but were masked out of the parent. This catches cases the physical-count check deliberately skips (e.g. torch-derived counts).

Source

Thrown at studio/backend/utils/hardware/hardware.py:2813

    # Only enforce the physical upper bound when the count is reliable (nvidia-smi).
    # A torch count reflects only visible devices, so it could falsely reject valid
    # physical indices. The parent-visible check below is always authoritative.
    if physical_gpu_count > 0 and parent_visible_ids:
        max_parent_id = max(parent_visible_ids)
        if physical_gpu_count > max_parent_id:
            # Count is plausibly physical, so enforce it.
            out_of_range = [gpu_id for gpu_id in requested_ids if gpu_id >= physical_gpu_count]
            if out_of_range:
                raise ValueError(
                    f"Invalid gpu_ids {requested_ids}: IDs must be physical GPU IDs "
                    f"between 0 and {physical_gpu_count - 1}. "
                    f"Rejected IDs: {out_of_range}. Parent-visible GPUs: {parent_visible_ids}"
                )

    disallowed_ids = [gpu_id for gpu_id in requested_ids if gpu_id not in parent_visible_ids]
    if disallowed_ids:
        raise ValueError(
            f"Invalid gpu_ids {requested_ids}: requested GPUs {disallowed_ids} are "
            f"outside the parent-visible set {parent_visible_ids}"
        )

    return requested_ids


def _resolve_model_identifier_for_gpu_estimate(
    model_name: str, hf_token: Optional[str] = None
) -> str:
    try:
        from utils.models.model_config import ModelConfig

        config = ModelConfig.from_identifier(model_name, hf_token = hf_token)
        if config and config.is_lora and config.base_model:
            return config.base_model
        return config.identifier if config else model_name
    except Exception as e:

View on GitHub (pinned to 203007d190)

Solutions

1. Restrict requested gpu_ids to the parent-visible set listed in the error message.
2. Or widen the parent's visibility: unset/fix CUDA_VISIBLE_DEVICES (or docker --gpus all) and restart the parent process so it can see the extra GPUs.
3. Verify from inside the parent process (nvidia-smi or torch.cuda.device_count()) what is actually visible before choosing ids.
4. With per-job GPU allocation (slurm/k8s), address GPUs by allocation-relative indices (usually 0-based within the visible set).

Example fix

# before
# parent launched with CUDA_VISIBLE_DEVICES=0,1
select_gpus([3])  # ValueError: outside parent-visible set [0, 1]

# after
select_gpus([0, 1])  # or relaunch parent without the mask to expose GPU 3

import os

def parent_visible_ids() -> list[int]:
    mask = os.environ.get("CUDA_VISIBLE_DEVICES")
    if not mask:
        return list(range(physical_gpu_count()))  # all visible
    return [int(p) for p in mask.split(",") if p.strip().isdigit()]

# guard: assert set(gpu_ids) <= set(parent_visible_ids())

try:
    resolved = resolve_requested_gpu_ids(gpu_ids)
except ValueError as e:
    if "parent-visible set" in str(e):
        gpu_ids = [i for i in gpu_ids if i in parent_visible_ids()]
        resolved = resolve_requested_gpu_ids(gpu_ids)
    else:
        raise

When it happens

Trigger: CUDA_VISIBLE_DEVICES=0,1 in the parent while requesting gpu_ids=[3]; or requesting an id hidden by a container/slurm GPU pin even though the machine physically has that GPU.

Common situations: Docker --gpus or k8s device plugins narrowing visibility below machine capacity; slurm allocating a GPU subset; a shell export of CUDA_VISIBLE_DEVICES from a previous session still active; mismatch between the shell that launched the parent and the one computing ids.

Related errors

• Invalid gpu_ids {requested_ids}: explicit physical GPU IDs a
• GPU selection is unavailable on this host: {exc}
• The current inference worker did not exit and still holds GP
• torch crashes when allocating on {device}; this install's to
• Invalid gpu_ids {requested_ids}: duplicate GPU IDs are not a

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/645d0c4524f36429. Report an issue: GitHub.