Lightning-AI/pytorch-lightning · critical · RuntimeError

The selected device indices {selected_device_indices!r} don'

Error message

The selected device indices {selected_device_indices!r} don't match the local rank values of processes. If you need to select GPUs at a specific index, set the `CUDA_VISIBLE_DEVICES` environment variable instead. For example: `CUDA_VISIBLE_DEVICES={','.join(str(i) for i in selected_device_indices)}`.

What it means

DeepSpeed maps each process's local rank to CUDA device index = local rank. _validate_device_index_selection requires parallel_devices to be exactly [cuda:0, cuda:1, ...] in order; otherwise processes would attach to the wrong GPUs. Selecting arbitrary indices must instead be done with CUDA_VISIBLE_DEVICES.

Source

Thrown at src/lightning/fabric/strategies/deepspeed.py:883

        "dp_world_size",
        "mp_world_size",
        "ds_config",
        "ds_version",
    }
    colliding_keys = deepspeed_internal_keys.intersection(state.keys())
    if colliding_keys:
        rank_zero_warn(
            "Your state has keys that collide with DeepSpeed's internal engine state. This could result in your"
            " values being overwritten by DeepSpeed. Consider changing the name of these keys to something else: "
            + ", ".join(colliding_keys)
        )


def _validate_device_index_selection(parallel_devices: list[torch.device]) -> None:
    selected_device_indices = [device.index for device in parallel_devices]
    expected_device_indices = list(range(len(parallel_devices)))
    if selected_device_indices != expected_device_indices:
        raise RuntimeError(
            f"The selected device indices {selected_device_indices!r} don't match the local rank values of processes."
            " If you need to select GPUs at a specific index, set the `CUDA_VISIBLE_DEVICES` environment variable"
            f" instead. For example: `CUDA_VISIBLE_DEVICES={','.join(str(i) for i in selected_device_indices)}`."
        )


def _is_deepspeed_checkpoint(path: str, fs: "AbstractFileSystem") -> bool:
    """Heuristic check whether the path points to a top-level DeepSpeed checkpoint directory."""
    return fs.isdir(path) and fs.isdir(f"{path.rstrip('/')}/checkpoint")


def _validate_checkpoint_directory(path: _PATH) -> None:
    """Validates that the path points to a DeepSpeed checkpoint directory and suggests fixes for user error."""
    # Example DeepSpeed checkpoint directory:
    #
    # epoch=5-step=10999.ckpt
    # ├── checkpoint
    # │   ├── zero_pp_rank_0_mp_rank_00_model_states.pt

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Set CUDA_VISIBLE_DEVICES=2,3 (your selected indices) before launching and pass contiguous devices (or let Lightning auto-detect)
2. Pass parallel_devices=[torch.device('cuda', i) for i in range(num_gpus)] so indices start at 0 and are contiguous
3. Wrap device selection in a launcher script that exports CUDA_VISIBLE_DEVICES

Example fix

# before
strategy = DeepSpeedStrategy(
    parallel_devices=[torch.device('cuda', 2), torch.device('cuda', 3)]
)

# after
# launch with: CUDA_VISIBLE_DEVICES=2,3 python train.py
strategy = DeepSpeedStrategy(
    parallel_devices=[torch.device('cuda', 0), torch.device('cuda', 1)]
)

indices = [d.index for d in parallel_devices]
assert indices == list(range(len(indices))), (
    "non-contiguous CUDA indices; set CUDA_VISIBLE_DEVICES instead"
)
strategy = DeepSpeedStrategy(parallel_devices=parallel_devices)

def valid_ds_devices(devices: list[torch.device]) -> bool:
    return [d.index for d in devices] == list(range(len(devices)))

When it happens

Trigger: Passing parallel_devices=[torch.device('cuda', 1), torch.device('cuda', 2)] (a subset or reordered list) to DeepSpeedStrategy; commonly produced by filtering torch.cuda.device_count() devices or using Lightning'sDevicesSelector with non-contiguous indices.

Common situations: Trying to run on GPUs 2-3 of an 8-GPU node by slicing parallel_devices; reordering devices to skip a bad GPU; adapting multi-device code from DDPStrategy where indices are respected.

Related errors

• The DeepSpeed strategy is only supported on CUDA GPUs but `{
• The DeepSpeed strategy is only supported on CUDA GPUs but `{
• No models were set up for backward. Did you forget to call `
• When using multiple models + deepspeed, please provide the m
• The `{type(self._strategy).__name__}` requires the model and

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/46fac214a5c01d00. Report an issue: GitHub.