Lightning-AI/pytorch-lightning · error · RuntimeError

Lightning attempted to launch new distributed processes with

Error message

Lightning attempted to launch new distributed processes with `local_rank > 0`. This should not happen. Possible reasons: 1) LOCAL_RANK environment variable was incorrectly modified by the user, 2) `ClusterEnvironment.creates_processes_externally` incorrectly implemented.

What it means

_SubprocessScriptLauncher is only supposed to spawn worker processes from the process with local rank 0. If cluster_environment.local_rank() != 0 at spawn time, something is wrong: the LOCAL_RANK env var was tampered with, or a custom ClusterEnvironment implementing creates_processes_externally incorrectly let every rank try to launch.

Source

Thrown at src/lightning/pytorch/strategies/launchers/subprocess_script.py:155

            cwd: Optional[str] = None
            if _HYDRA_AVAILABLE:
                from hydra.core.hydra_config import HydraConfig

                hydra_in_use = HydraConfig.initialized()

            if hydra_in_use:
                command, cwd = _hydra_subprocess_cmd(local_rank)
            else:
                command = _basic_subprocess_cmd()

            new_process = subprocess.Popen(command, env=env_copy, cwd=cwd)
            self.procs.append(new_process)

    def _check_can_spawn_children(self) -> None:
        if len(self.procs) > 0:
            raise RuntimeError("The launcher can only create subprocesses once.")
        if self.cluster_environment.local_rank() != 0:
            raise RuntimeError(
                "Lightning attempted to launch new distributed processes with `local_rank > 0`. This should not happen."
                " Possible reasons: 1) LOCAL_RANK environment variable was incorrectly modified by the user,"
                " 2) `ClusterEnvironment.creates_processes_externally` incorrectly implemented."
            )

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Unset leftover launcher env vars (LOCAL_RANK, RANK, WORLD_SIZE, etc.) before running standalone: e.g. env -u LOCAL_RANK python train.py
2. If using torchrun/torchelastic, use the strategy/launcher designed for external process creation (creates_processes_externally=True, e.g. _SubprocessScriptLauncher via XLAStrategy/defaults)
3. Fix custom ClusterEnvironment.creates_processes_externally and local_rank() to report the truth

Example fix

# before
# shell still has torchrun vars
LOCAL_RANK=1 python train.py  # RuntimeError

# after
env -u LOCAL_RANK -u RANK -u WORLD_SIZE python train.py

import os
assert int(os.environ.get("LOCAL_RANK", 0)) == 0, "run from rank-0 process or unset LOCAL_RANK"

When it happens

Trigger: Setting/modifying the LOCAL_RANK environment variable manually before running the script; using a custom ClusterEnvironment whose creates_processes_externally returns False while ranks other than 0 reach _check_can_spawn_children; launching via a tool that exports LOCAL_RANK>0 (e.g. torchrun leftover env) while using a strategy that spawns its own processes.

Common situations: Switching a script between torchrun (external spawning) and Lightning-internal spawning without clearing LOCAL_RANK/related env vars; buggy custom cluster environments in house-made plugins.

Related errors

• The Kubeflow environment can't be detected automatically.
• Lightning attempted to launch new distributed processes with
• The launcher can only create subprocesses once.
• cluster_environment set through both strategy class and plug
• `num_nodes` must be a positive integer, but got {num_nodes}.

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/df4ce323e297884a. Report an issue: GitHub.