Lightning-AI/pytorch-lightning · error · RuntimeError

The launcher can only create subprocesses once.

Error message

The launcher can only create subprocesses once.

What it means

_SubprocessScriptLauncher spawns all distributed worker processes once, from rank 0, by re-executing the user's script. It records spawned processes in self.procs and refuses to spawn again if that list is non-empty, because the launcher is designed for a single launch per run.

Source

Thrown at src/lightning/pytorch/strategies/launchers/subprocess_script.py:153

            hydra_in_use = False
            cwd: Optional[str] = None
            if _HYDRA_AVAILABLE:
                from hydra.core.hydra_config import HydraConfig

                hydra_in_use = HydraConfig.initialized()

            if hydra_in_use:
                command, cwd = _hydra_subprocess_cmd(local_rank)
            else:
                command = _basic_subprocess_cmd()

            new_process = subprocess.Popen(command, env=env_copy, cwd=cwd)
            self.procs.append(new_process)

    def _check_can_spawn_children(self) -> None:
        if len(self.procs) > 0:
            raise RuntimeError("The launcher can only create subprocesses once.")
        if self.cluster_environment.local_rank() != 0:
            raise RuntimeError(
                "Lightning attempted to launch new distributed processes with `local_rank > 0`. This should not happen."
                " Possible reasons: 1) LOCAL_RANK environment variable was incorrectly modified by the user,"
                " 2) `ClusterEnvironment.creates_processes_externally` incorrectly implemented."
            )

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Ensure the launcher only launches once: subsequent phases (test/validate/predict) should reuse existing processes or a new launcher instance is created by a fresh Trainer/strategy
2. Restructure the script so all distributed work happens within one launch closure
3. Check you are not manually calling strategy.launcher.launch in custom code

Example fix

# before
launcher.launch(fit_fn)
launcher.launch(test_fn)  # RuntimeError

# after
launcher.launch(lambda: (fit_fn(), test_fn()))  # single launch for all distributed work

assert not getattr(launcher, "procs", None), "launcher already spawned once"

When it happens

Trigger: Calling launcher.launch() twice on the same _SubprocessScriptLauncher instance, e.g. invoking fit() and then test()/validate() in a way that re-enters the launcher within one process, with self.procs already populated.

Common situations: Running fit() followed by a second distributed entry point in the same script with a strategy whose launcher state persisted; custom code that manually invokes the launcher's launch more than once.

Related errors

• The launcher can only create subprocesses once.
• Lightning attempted to launch new distributed processes with
• `{type(self).__name__}` does not own a group. HINT: try `col
• `{type(self).__name__}` already owns a group.
• `{type(self).__name__}` does not own a group to destroy.

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/698bd812a432635f. Report an issue: GitHub.