Lightning-AI/pytorch-lightning · critical · RuntimeError

torch.distributed is not available. Cannot initialize distri

Error message

torch.distributed is not available. Cannot initialize distributed process group

What it means

Lightning's distributed setup calls _init_dist_connection to create the torch.distributed process group; if torch.distributed.is_available() is False (PyTorch built without distributed support, e.g. Windows builds or stripped wheels), it raises RuntimeError before any rendezvous happens. It is a build/dependency problem, not a topology problem.

Source

Thrown at src/lightning/fabric/utilities/distributed.py:273

    **kwargs: Any,
) -> None:
    """Utility function to initialize distributed connection by setting env variables and initializing the distributed
    process group.

    Args:
        cluster_environment: ``ClusterEnvironment`` instance
        torch_distributed_backend: Backend to use (includes `nccl` and `gloo`)
        global_rank: Rank of the current process
        world_size: Number of processes in the group
        kwargs: Kwargs for ``init_process_group``

    Raises:
        RuntimeError:
            If ``torch.distributed`` is not available

    """
    if not torch.distributed.is_available():
        raise RuntimeError("torch.distributed is not available. Cannot initialize distributed process group")
    if torch.distributed.is_initialized():
        log.debug("torch.distributed is already initialized. Exiting early")
        return
    global_rank = global_rank if global_rank is not None else cluster_environment.global_rank()
    world_size = world_size if world_size is not None else cluster_environment.world_size()
    os.environ["MASTER_ADDR"] = cluster_environment.main_address
    os.environ["MASTER_PORT"] = str(cluster_environment.main_port)
    log.info(f"Initializing distributed: GLOBAL_RANK: {global_rank}, MEMBER: {global_rank + 1}/{world_size}")
    torch.distributed.init_process_group(torch_distributed_backend, rank=global_rank, world_size=world_size, **kwargs)

    if torch_distributed_backend == "nccl":
        # PyTorch >= 2.4 warns about undestroyed NCCL process group, so we need to do it at program exit
        atexit.register(_destroy_dist_connection)

    # On rank=0 let everyone know training is starting
    rank_zero_info(
        f"{'-' * 100}\n"
        f"distributed_backend={torch_distributed_backend}\n"

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Verify availability: python -c "import torch; print(torch.distributed.is_available())"
2. Install a PyTorch build with distributed support (e.g. the default Linux CUDA/CPU wheels, or use WSL2 on Windows)
3. If distributed isn't needed, use a single device or strategy='ddp' replaced by non-distributed setup (devices=1 / no parallel strategy)

Example fix

# before
fabric = Fabric(strategy="ddp", devices=2)  # torch.distributed unavailable

# after (WSL/Linux with proper torch), or:
fabric = Fabric(devices=1)  # no process group needed

import torch

if not torch.distributed.is_available():
    raise SystemExit("This PyTorch build lacks torch.distributed; reinstall a full wheel or run single-device")

When it happens

Trigger: Running multi-device Fabric/Trainer (devices>1 or a distributed strategy) on a PyTorch build lacking torch.distributed; some pip Windows wheels and old conda builds; using parallel strategy in an environment where 'python -c "import torch.distributed"' fails.

Common situations: Windows laptops trying DDP without WSL; minimal/embedded PyTorch wheels; version mismatches after upgrading torch; CI images with CPU-only builds.

Related errors

• Post-localSGD algorithm is used, but model averaging period
• `num_nodes` must be a positive integer, but got {num_nodes}.
• To use Fabric with more than one device, you must call `.lau
• `{type(self).__name__}` does not own a group. HINT: try `col
• `{type(self).__name__}` already owns a group.

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/0d95585d59cfa110. Report an issue: GitHub.