unslothai/unsloth · error · ValueError

Invalid gpu_ids {requested_ids}: IDs must be physical GPU ID

Error message

Invalid gpu_ids {requested_ids}: IDs must be physical GPU IDs between 0 and {physical_gpu_count - 1}. Rejected IDs: {out_of_range}. Parent-visible GPUs: {parent_visible_ids}

What it means

Raised when a requested id is >= physical_gpu_count, but only when the count is trusted (sourced from nvidia-smi and plausibly physical: count > max parent-visible id). This guard prevents silently selecting nonexistent devices when torch's visible-only count would under-report. The message states the valid 0..count-1 range, the rejected ids, and the parent-visible set.

Source

Thrown at studio/backend/utils/hardware/hardware.py:2805

    # Reject negative IDs.
    negative_ids = [gpu_id for gpu_id in requested_ids if gpu_id < 0]
    if negative_ids:
        raise ValueError(
            f"Invalid gpu_ids {requested_ids}: GPU IDs must be non-negative. "
            f"Rejected IDs: {negative_ids}. Parent-visible GPUs: {parent_visible_ids}"
        )

    # Only enforce the physical upper bound when the count is reliable (nvidia-smi).
    # A torch count reflects only visible devices, so it could falsely reject valid
    # physical indices. The parent-visible check below is always authoritative.
    if physical_gpu_count > 0 and parent_visible_ids:
        max_parent_id = max(parent_visible_ids)
        if physical_gpu_count > max_parent_id:
            # Count is plausibly physical, so enforce it.
            out_of_range = [gpu_id for gpu_id in requested_ids if gpu_id >= physical_gpu_count]
            if out_of_range:
                raise ValueError(
                    f"Invalid gpu_ids {requested_ids}: IDs must be physical GPU IDs "
                    f"between 0 and {physical_gpu_count - 1}. "
                    f"Rejected IDs: {out_of_range}. Parent-visible GPUs: {parent_visible_ids}"
                )

    disallowed_ids = [gpu_id for gpu_id in requested_ids if gpu_id not in parent_visible_ids]
    if disallowed_ids:
        raise ValueError(
            f"Invalid gpu_ids {requested_ids}: requested GPUs {disallowed_ids} are "
            f"outside the parent-visible set {parent_visible_ids}"
        )

    return requested_ids


def _resolve_model_identifier_for_gpu_estimate(
    model_name: str, hf_token: Optional[str] = None
) -> str:

View on GitHub (pinned to 203007d190)

Solutions

1. Use ids in [0, physical_gpu_count-1]; check nvidia-smi -L for the real count and ordinals.
2. If the machine genuinely has more GPUs, fix CUDA_VISIBLE_DEVICES hiding them and restart the parent process (masks are read at parent startup).
3. Prefer auto-selection (omit gpu_ids) unless specific placement is required.
4. Parameterize configs by hostname/GPU count instead of hardcoding id lists.

Example fix

# before
select_gpus([2])  # 2-GPU machine -> ValueError (valid: 0..1)

# after
select_gpus([0, 1])  # or omit ids for auto-select

import subprocess

def physical_gpu_count() -> int:
    out = subprocess.run(
        ["nvidia-smi", "-L"], capture_output=True, text=True
    ).stdout
    return sum(1 for line in out.splitlines() if line.startswith("GPU "))

# guard: assert all(0 <= i < physical_gpu_count() for i in gpu_ids)

When it happens

Trigger: Requesting gpu_ids=[2] on a 2-GPU machine (valid ids 0..1) where nvidia-smi reports physical_gpu_count=2; requesting an id valid on another machine after moving a config.

Common situations: Config written on an 8-GPU box reused on a 2-GPU box; misunderstanding that ids are physical indices not per-job slots; off-by-one from treating count as the last index.

Related errors

• Invalid gpu_ids {requested_ids}: duplicate GPU IDs are not a
• Invalid gpu_ids {requested_ids}: GPU IDs must be non-negativ
• GPU selection is unavailable on this host: {exc}
• The current inference worker did not exit and still holds GP
• torch crashes when allocating on {device}; this install's to

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/27ab1e9bdac090ea. Report an issue: GitHub.