unslothai/unsloth · error · ValueError

Invalid gpu_ids {requested_ids}: explicit physical GPU IDs a

Error message

Invalid gpu_ids {requested_ids}: explicit physical GPU IDs are unsupported when {env_var_name} uses non-numeric or subdevice entries ({parent_visible_spec['raw']!r}). Omit gpu_ids to use the parent-visible devices.

What it means

Raised on the CUDA/ROCm/XPU path when explicit gpu_ids are requested but the parent process's visibility env var (CUDA_VISIBLE_DEVICES or ZE_AFFINITY_MASK on XPU) contains non-numeric or subdevice entries, so physical ids in the mask cannot be reliably mapped. The raw mask value is included in the message. The fix is to omit gpu_ids and inherit the parent-visible set.

Source

Thrown at studio/backend/utils/hardware/hardware.py:2775

        # A Vulkan build selects by ggml Vulkan ordinal (--device VulkanN), a separate
        # index space from CUDA/ROCm ids that may be empty under CPU-only torch. The
        # CUDA parent-visible / physical-count checks below do not apply; only reject
        # malformed ordinals (issue #7239).
        if len(set(requested_ids)) != len(requested_ids):
            raise ValueError(f"Invalid gpu_ids {requested_ids}: duplicate GPU IDs are not allowed.")
        negative_ids = [gpu_id for gpu_id in requested_ids if gpu_id < 0]
        if negative_ids:
            raise ValueError(
                f"Invalid gpu_ids {requested_ids}: GPU IDs must be non-negative. "
                f"Rejected IDs: {negative_ids}."
            )
        return requested_ids

    if not parent_visible_spec["supports_explicit_gpu_ids"]:
        env_var_name = (
            "ZE_AFFINITY_MASK" if get_device() == DeviceType.XPU else "CUDA_VISIBLE_DEVICES"
        )
        raise ValueError(
            f"Invalid gpu_ids {requested_ids}: explicit physical GPU IDs are "
            f"unsupported when {env_var_name} uses non-numeric or subdevice "
            f"entries ({parent_visible_spec['raw']!r}). Omit gpu_ids to use "
            "the parent-visible devices."
        )

    if len(set(requested_ids)) != len(requested_ids):
        raise ValueError(
            f"Invalid gpu_ids {requested_ids}: duplicate GPU IDs are not allowed. "
            f"Parent-visible GPUs: {parent_visible_ids}"
        )

    # Reject negative IDs.
    negative_ids = [gpu_id for gpu_id in requested_ids if gpu_id < 0]
    if negative_ids:
        raise ValueError(
            f"Invalid gpu_ids {requested_ids}: GPU IDs must be non-negative. "
            f"Rejected IDs: {negative_ids}. Parent-visible GPUs: {parent_visible_ids}"

View on GitHub (pinned to 203007d190)

Solutions

1. Omit gpu_ids and rely on the parent-visible devices already narrowed by the mask.
2. Or set CUDA_VISIBLE_DEVICES to plain numeric ids (e.g. '0,1') before launching so explicit selection works.
3. On XPU, simplify ZE_AFFINITY_MASK to whole-device numeric form.
4. If you must keep UUID masks, translate them to ordinals via nvidia-smi -L ordering first.

Example fix

# before
CUDA_VISIBLE_DEVICES=GPU-aa11bb22... python app.py --gpu-ids 0  # ValueError

# after
CUDA_VISIBLE_DEVICES=GPU-aa11bb22... python app.py  # omit gpu_ids, inherit mask

import os, re

def mask_supports_explicit_ids() -> bool:
    """CUDA_VISIBLE_DEVICES must be numeric-only (or unset) for explicit gpu_ids."""
    mask = os.environ.get("CUDA_VISIBLE_DEVICES", "")
    if not mask:
        return True
    return all(re.fullmatch(r"\d+", part) for part in mask.split(","))

# if requesting explicit ids: assert mask_supports_explicit_ids()

try:
    resolved = resolve_requested_gpu_ids(gpu_ids)
except ValueError as e:
    if "Omit gpu_ids" in str(e):
        resolved, meta = select_gpu_ids(None)  # inherit parent-visible devices
    else:
        raise

When it happens

Trigger: Setting CUDA_VISIBLE_DEVICES to a value like 'GPU-abcdef,MIG-...' (UUIDs) or ZE_AFFINITY_MASK with subdevice syntax like '0.1', then requesting explicit numeric gpu_ids.

Common situations: Running inside containers/slurm that pin GPUs by UUID; MIG partitions exposing MIG-uuid masks; Intel oneAPI setups using ZE_AFFINITY_MASK=device.subdevice; users copying GPU-uuid masks from nvidia-smi -L into the env var.

Related errors

• Invalid gpu_ids {requested_ids}: requested GPUs {disallowed_
• GPU selection is unavailable on this host: {exc}
• The current inference worker did not exit and still holds GP
• torch crashes when allocating on {device}; this install's to
• Invalid gpu_ids {requested_ids}: duplicate GPU IDs are not a

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/85c06fcd72308b23. Report an issue: GitHub.