{"record":{"id":"85c06fcd72308b23","repo":"unslothai/unsloth","slug":"invalid-gpu-ids-requested-ids-explicit-physical","errorCode":null,"errorMessage":"Invalid gpu_ids {requested_ids}: explicit physical GPU IDs are unsupported when {env_var_name} uses non-numeric or subdevice entries ({parent_visible_spec['raw']!r}). Omit gpu_ids to use the parent-visible devices.","messagePattern":"Invalid gpu_ids (.+?): explicit physical GPU IDs are unsupported when (.+?) uses non-numeric or subdevice entries \\((.+?)\\)\\. Omit gpu_ids to use the parent-visible devices\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"studio/backend/utils/hardware/hardware.py","lineNumber":2775,"sourceCode":"        # A Vulkan build selects by ggml Vulkan ordinal (--device VulkanN), a separate\n        # index space from CUDA/ROCm ids that may be empty under CPU-only torch. The\n        # CUDA parent-visible / physical-count checks below do not apply; only reject\n        # malformed ordinals (issue #7239).\n        if len(set(requested_ids)) != len(requested_ids):\n            raise ValueError(f\"Invalid gpu_ids {requested_ids}: duplicate GPU IDs are not allowed.\")\n        negative_ids = [gpu_id for gpu_id in requested_ids if gpu_id < 0]\n        if negative_ids:\n            raise ValueError(\n                f\"Invalid gpu_ids {requested_ids}: GPU IDs must be non-negative. \"\n                f\"Rejected IDs: {negative_ids}.\"\n            )\n        return requested_ids\n\n    if not parent_visible_spec[\"supports_explicit_gpu_ids\"]:\n        env_var_name = (\n            \"ZE_AFFINITY_MASK\" if get_device() == DeviceType.XPU else \"CUDA_VISIBLE_DEVICES\"\n        )\n        raise ValueError(\n            f\"Invalid gpu_ids {requested_ids}: explicit physical GPU IDs are \"\n            f\"unsupported when {env_var_name} uses non-numeric or subdevice \"\n            f\"entries ({parent_visible_spec['raw']!r}). Omit gpu_ids to use \"\n            \"the parent-visible devices.\"\n        )\n\n    if len(set(requested_ids)) != len(requested_ids):\n        raise ValueError(\n            f\"Invalid gpu_ids {requested_ids}: duplicate GPU IDs are not allowed. \"\n            f\"Parent-visible GPUs: {parent_visible_ids}\"\n        )\n\n    # Reject negative IDs.\n    negative_ids = [gpu_id for gpu_id in requested_ids if gpu_id < 0]\n    if negative_ids:\n        raise ValueError(\n            f\"Invalid gpu_ids {requested_ids}: GPU IDs must be non-negative. \"\n            f\"Rejected IDs: {negative_ids}. Parent-visible GPUs: {parent_visible_ids}\"","sourceCodeStart":2757,"sourceCodeEnd":2793,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/utils/hardware/hardware.py#L2757-L2793","documentation":"Raised on the CUDA/ROCm/XPU path when explicit gpu_ids are requested but the parent process's visibility env var (CUDA_VISIBLE_DEVICES or ZE_AFFINITY_MASK on XPU) contains non-numeric or subdevice entries, so physical ids in the mask cannot be reliably mapped. The raw mask value is included in the message. The fix is to omit gpu_ids and inherit the parent-visible set.","triggerScenarios":"Setting CUDA_VISIBLE_DEVICES to a value like 'GPU-abcdef,MIG-...' (UUIDs) or ZE_AFFINITY_MASK with subdevice syntax like '0.1', then requesting explicit numeric gpu_ids.","commonSituations":"Running inside containers/slurm that pin GPUs by UUID; MIG partitions exposing MIG-uuid masks; Intel oneAPI setups using ZE_AFFINITY_MASK=device.subdevice; users copying GPU-uuid masks from nvidia-smi -L into the env var.","solutions":["Omit gpu_ids and rely on the parent-visible devices already narrowed by the mask.","Or set CUDA_VISIBLE_DEVICES to plain numeric ids (e.g. '0,1') before launching so explicit selection works.","On XPU, simplify ZE_AFFINITY_MASK to whole-device numeric form.","If you must keep UUID masks, translate them to ordinals via nvidia-smi -L ordering first."],"exampleFix":"# before\nCUDA_VISIBLE_DEVICES=GPU-aa11bb22... python app.py --gpu-ids 0  # ValueError\n\n# after\nCUDA_VISIBLE_DEVICES=GPU-aa11bb22... python app.py  # omit gpu_ids, inherit mask","handlingStrategy":"validation","validationCode":"import os, re\n\ndef mask_supports_explicit_ids() -> bool:\n    \"\"\"CUDA_VISIBLE_DEVICES must be numeric-only (or unset) for explicit gpu_ids.\"\"\"\n    mask = os.environ.get(\"CUDA_VISIBLE_DEVICES\", \"\")\n    if not mask:\n        return True\n    return all(re.fullmatch(r\"\\d+\", part) for part in mask.split(\",\"))\n\n# if requesting explicit ids: assert mask_supports_explicit_ids()","typeGuard":null,"tryCatchPattern":"try:\n    resolved = resolve_requested_gpu_ids(gpu_ids)\nexcept ValueError as e:\n    if \"Omit gpu_ids\" in str(e):\n        resolved, meta = select_gpu_ids(None)  # inherit parent-visible devices\n    else:\n        raise","preventionTips":["Keep CUDA_VISIBLE_DEVICES numeric ('0,1') when you need explicit id selection.","In containers/slurm that pin by UUID or MIG, omit gpu_ids and inherit the mask.","On Intel, prefer whole-device ZE_AFFINITY_MASK form ('0,1' not '0.1').","Log the effective mask at startup to catch environment surprises early."],"tags":["gpu","cuda","xpu","env-vars","containers"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}