{"record":{"id":"46fac214a5c01d00","repo":"Lightning-AI/pytorch-lightning","slug":"the-selected-device-indices-selected-device-indic","errorCode":null,"errorMessage":"The selected device indices {selected_device_indices!r} don't match the local rank values of processes. If you need to select GPUs at a specific index, set the `CUDA_VISIBLE_DEVICES` environment variable instead. For example: `CUDA_VISIBLE_DEVICES={','.join(str(i) for i in selected_device_indices)}`.","messagePattern":"The selected device indices (.+?) don't match the local rank values of processes\\. If you need to select GPUs at a specific index, set the `CUDA_VISIBLE_DEVICES` environment variable instead\\. For example: `CUDA_VISIBLE_DEVICES=(.+?)`\\.","errorType":"error_code","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"src/lightning/fabric/strategies/deepspeed.py","lineNumber":883,"sourceCode":"        \"dp_world_size\",\n        \"mp_world_size\",\n        \"ds_config\",\n        \"ds_version\",\n    }\n    colliding_keys = deepspeed_internal_keys.intersection(state.keys())\n    if colliding_keys:\n        rank_zero_warn(\n            \"Your state has keys that collide with DeepSpeed's internal engine state. This could result in your\"\n            \" values being overwritten by DeepSpeed. Consider changing the name of these keys to something else: \"\n            + \", \".join(colliding_keys)\n        )\n\n\ndef _validate_device_index_selection(parallel_devices: list[torch.device]) -> None:\n    selected_device_indices = [device.index for device in parallel_devices]\n    expected_device_indices = list(range(len(parallel_devices)))\n    if selected_device_indices != expected_device_indices:\n        raise RuntimeError(\n            f\"The selected device indices {selected_device_indices!r} don't match the local rank values of processes.\"\n            \" If you need to select GPUs at a specific index, set the `CUDA_VISIBLE_DEVICES` environment variable\"\n            f\" instead. For example: `CUDA_VISIBLE_DEVICES={','.join(str(i) for i in selected_device_indices)}`.\"\n        )\n\n\ndef _is_deepspeed_checkpoint(path: str, fs: \"AbstractFileSystem\") -> bool:\n    \"\"\"Heuristic check whether the path points to a top-level DeepSpeed checkpoint directory.\"\"\"\n    return fs.isdir(path) and fs.isdir(f\"{path.rstrip('/')}/checkpoint\")\n\n\ndef _validate_checkpoint_directory(path: _PATH) -> None:\n    \"\"\"Validates that the path points to a DeepSpeed checkpoint directory and suggests fixes for user error.\"\"\"\n    # Example DeepSpeed checkpoint directory:\n    #\n    # epoch=5-step=10999.ckpt\n    # ├── checkpoint\n    # │   ├── zero_pp_rank_0_mp_rank_00_model_states.pt","sourceCodeStart":865,"sourceCodeEnd":901,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/fabric/strategies/deepspeed.py#L865-L901","documentation":"DeepSpeed maps each process's local rank to CUDA device index = local rank. _validate_device_index_selection requires parallel_devices to be exactly [cuda:0, cuda:1, ...] in order; otherwise processes would attach to the wrong GPUs. Selecting arbitrary indices must instead be done with CUDA_VISIBLE_DEVICES.","triggerScenarios":"Passing parallel_devices=[torch.device('cuda', 1), torch.device('cuda', 2)] (a subset or reordered list) to DeepSpeedStrategy; commonly produced by filtering torch.cuda.device_count() devices or using Lightning'sDevicesSelector with non-contiguous indices.","commonSituations":"Trying to run on GPUs 2-3 of an 8-GPU node by slicing parallel_devices; reordering devices to skip a bad GPU; adapting multi-device code from DDPStrategy where indices are respected.","solutions":["Set CUDA_VISIBLE_DEVICES=2,3 (your selected indices) before launching and pass contiguous devices (or let Lightning auto-detect)","Pass parallel_devices=[torch.device('cuda', i) for i in range(num_gpus)] so indices start at 0 and are contiguous","Wrap device selection in a launcher script that exports CUDA_VISIBLE_DEVICES"],"exampleFix":"# before\nstrategy = DeepSpeedStrategy(\n    parallel_devices=[torch.device('cuda', 2), torch.device('cuda', 3)]\n)\n\n# after\n# launch with: CUDA_VISIBLE_DEVICES=2,3 python train.py\nstrategy = DeepSpeedStrategy(\n    parallel_devices=[torch.device('cuda', 0), torch.device('cuda', 1)]\n)","handlingStrategy":"validation","validationCode":"indices = [d.index for d in parallel_devices]\nassert indices == list(range(len(indices))), (\n    \"non-contiguous CUDA indices; set CUDA_VISIBLE_DEVICES instead\"\n)\nstrategy = DeepSpeedStrategy(parallel_devices=parallel_devices)","typeGuard":"def valid_ds_devices(devices: list[torch.device]) -> bool:\n    return [d.index for d in devices] == list(range(len(devices)))","tryCatchPattern":null,"preventionTips":["Select GPUs only via CUDA_VISIBLE_DEVICES","Never slice/reorder parallel_devices for DeepSpeed"],"tags":["deepspeed","cuda","device-selection","multi-gpu"],"backgroundTag":"gpu-device-selection-invalid","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}