{"record":{"id":"645d0c4524f36429","repo":"unslothai/unsloth","slug":"invalid-gpu-ids-requested-ids-requested-gpus-d","errorCode":null,"errorMessage":"Invalid gpu_ids {requested_ids}: requested GPUs {disallowed_ids} are outside the parent-visible set {parent_visible_ids}","messagePattern":"Invalid gpu_ids (.+?): requested GPUs (.+?) are outside the parent-visible set (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"studio/backend/utils/hardware/hardware.py","lineNumber":2813,"sourceCode":"\n    # Only enforce the physical upper bound when the count is reliable (nvidia-smi).\n    # A torch count reflects only visible devices, so it could falsely reject valid\n    # physical indices. The parent-visible check below is always authoritative.\n    if physical_gpu_count > 0 and parent_visible_ids:\n        max_parent_id = max(parent_visible_ids)\n        if physical_gpu_count > max_parent_id:\n            # Count is plausibly physical, so enforce it.\n            out_of_range = [gpu_id for gpu_id in requested_ids if gpu_id >= physical_gpu_count]\n            if out_of_range:\n                raise ValueError(\n                    f\"Invalid gpu_ids {requested_ids}: IDs must be physical GPU IDs \"\n                    f\"between 0 and {physical_gpu_count - 1}. \"\n                    f\"Rejected IDs: {out_of_range}. Parent-visible GPUs: {parent_visible_ids}\"\n                )\n\n    disallowed_ids = [gpu_id for gpu_id in requested_ids if gpu_id not in parent_visible_ids]\n    if disallowed_ids:\n        raise ValueError(\n            f\"Invalid gpu_ids {requested_ids}: requested GPUs {disallowed_ids} are \"\n            f\"outside the parent-visible set {parent_visible_ids}\"\n        )\n\n    return requested_ids\n\n\ndef _resolve_model_identifier_for_gpu_estimate(\n    model_name: str, hf_token: Optional[str] = None\n) -> str:\n    try:\n        from utils.models.model_config import ModelConfig\n\n        config = ModelConfig.from_identifier(model_name, hf_token = hf_token)\n        if config and config.is_lora and config.base_model:\n            return config.base_model\n        return config.identifier if config else model_name\n    except Exception as e:","sourceCodeStart":2795,"sourceCodeEnd":2831,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/utils/hardware/hardware.py#L2795-L2831","documentation":"The authoritative final GPU check: every requested id must be a member of the parent-visible set (devices the parent process can see after applying CUDA_VISIBLE_DEVICES/ZE_AFFINITY_MASK). It fires for ids that are valid physical ids but were masked out of the parent. This catches cases the physical-count check deliberately skips (e.g. torch-derived counts).","triggerScenarios":"CUDA_VISIBLE_DEVICES=0,1 in the parent while requesting gpu_ids=[3]; or requesting an id hidden by a container/slurm GPU pin even though the machine physically has that GPU.","commonSituations":"Docker --gpus or k8s device plugins narrowing visibility below machine capacity; slurm allocating a GPU subset; a shell export of CUDA_VISIBLE_DEVICES from a previous session still active; mismatch between the shell that launched the parent and the one computing ids.","solutions":["Restrict requested gpu_ids to the parent-visible set listed in the error message.","Or widen the parent's visibility: unset/fix CUDA_VISIBLE_DEVICES (or docker --gpus all) and restart the parent process so it can see the extra GPUs.","Verify from inside the parent process (nvidia-smi or torch.cuda.device_count()) what is actually visible before choosing ids.","With per-job GPU allocation (slurm/k8s), address GPUs by allocation-relative indices (usually 0-based within the visible set)."],"exampleFix":"# before\n# parent launched with CUDA_VISIBLE_DEVICES=0,1\nselect_gpus([3])  # ValueError: outside parent-visible set [0, 1]\n\n# after\nselect_gpus([0, 1])  # or relaunch parent without the mask to expose GPU 3","handlingStrategy":"validation","validationCode":"import os\n\ndef parent_visible_ids() -> list[int]:\n    mask = os.environ.get(\"CUDA_VISIBLE_DEVICES\")\n    if not mask:\n        return list(range(physical_gpu_count()))  # all visible\n    return [int(p) for p in mask.split(\",\") if p.strip().isdigit()]\n\n# guard: assert set(gpu_ids) <= set(parent_visible_ids())","typeGuard":null,"tryCatchPattern":"try:\n    resolved = resolve_requested_gpu_ids(gpu_ids)\nexcept ValueError as e:\n    if \"parent-visible set\" in str(e):\n        gpu_ids = [i for i in gpu_ids if i in parent_visible_ids()]\n        resolved = resolve_requested_gpu_ids(gpu_ids)\n    else:\n        raise","preventionTips":["Determine the visible set from inside the parent process, not your login shell.","In slurm/k8s/docker, map allocation-relative indices instead of host-wide ids.","Restart the parent after changing CUDA_VISIBLE_DEVICES — masks bind at spawn.","Log CUDA_VISIBLE_DEVICES and torch.cuda.device_count() at job start for diagnosis."],"tags":["gpu","cuda","env-vars","containers","scheduling"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}