Lightning-AI/pytorch-lightning · error · RuntimeError

Lightning attempted to launch new distributed processes with

Error message

Lightning attempted to launch new distributed processes with `local_rank > 0`. This should not happen. Possible reasons: 1) LOCAL_RANK environment variable was incorrectly modified by the user, 2) `ClusterEnvironment.creates_processes_externally` incorrectly implemented.

What it means

The subprocess-script launcher should only ever spawn children from the rank-0 process. If cluster_environment.local_rank() returns something other than 0 at spawn time, either the LOCAL_RANK env var was tampered with or the ClusterEnvironment reports creates_processes_externally incorrectly, and Lightning aborts to avoid every rank spawning a full copy of the job.

Source

Thrown at src/lightning/fabric/strategies/launchers/subprocess_script.py:151

            hydra_in_use = False
            cwd: Optional[str] = None
            if _HYDRA_AVAILABLE:
                from hydra.core.hydra_config import HydraConfig

                hydra_in_use = HydraConfig.initialized()
            if hydra_in_use:
                command, cwd = _hydra_subprocess_cmd(local_rank=local_rank)
            else:
                command = _basic_subprocess_cmd()

            proc = subprocess.Popen(command, env=env_copy, cwd=cwd)
            self.procs.append(proc)

    def _check_can_spawn_children(self) -> None:
        if len(self.procs) > 0:
            raise RuntimeError("The launcher can only create subprocesses once.")
        if self.cluster_environment.local_rank() != 0:
            raise RuntimeError(
                "Lightning attempted to launch new distributed processes with `local_rank > 0`. This should not happen."
                " Possible reasons: 1) LOCAL_RANK environment variable was incorrectly modified by the user,"
                " 2) `ClusterEnvironment.creates_processes_externally` incorrectly implemented."
            )


def _basic_subprocess_cmd() -> Sequence[str]:
    import __main__  # local import to avoid https://github.com/Lightning-AI/pytorch-lightning/issues/15218

    if __main__.__spec__ is None:  # pragma: no-cover
        return [sys.executable, os.path.abspath(sys.argv[0])] + sys.argv[1:]
    return [sys.executable, "-m", __main__.__spec__.name] + sys.argv[1:]


def _hydra_subprocess_cmd(local_rank: int) -> tuple[Sequence[str], str]:
    from hydra.core.hydra_config import HydraConfig
    from hydra.utils import get_original_cwd, to_absolute_path

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Unset LOCAL_RANK ('unset LOCAL_RANK' / remove it from env) and launch with plain python or fabric's CLI so Lightning spawns ranks itself
2. If you intend torchrun/srun to create processes, use the cluster environment/strategy designed for external launching (e.g. 'ddp' with creates_processes_externally=True)
3. Audit custom ClusterEnvironment implementations so local_rank() and creates_processes_externally agree

Example fix

# before
LOCAL_RANK=1 python train.py --strategy ddp

# after
unset LOCAL_RANK
python train.py --strategy ddp  # Lightning spawns all ranks from rank 0

import os
if int(os.environ.get("LOCAL_RANK", "0")) != 0 and launcher_is_subprocess_script:
    raise RuntimeError("LOCAL_RANK must be unset/0 for Lightning-spawned runs")

When it happens

Trigger: Manually setting/exporting LOCAL_RANK (or PL_GLOBAL_RANK-related vars) before running the script; using a custom ClusterEnvironment whose creates_processes_externally/local_rank are inconsistent; double-launching under an existing torchrun session.

Common situations: Running 'LOCAL_RANK=2 python train.py' by mistake; a wrapper script or scheduler injecting LOCAL_RANK; copying a launch command from a torchrun context into a plain python invocation; custom cluster plugins with wrong external-process flags.

Related errors

• The launcher can only create subprocesses once.
• `{type(self).__name__}` does not own a group. HINT: try `col
• `{type(self).__name__}` already owns a group.
• `{type(self).__name__}` does not own a group to destroy.
• Torch distributed is not available.

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/d8d2bbd2c1141082. Report an issue: GitHub.