Lightning-AI/pytorch-lightning · error · RuntimeError

The launcher can only create subprocesses once.

Error message

The launcher can only create subprocesses once.

What it means

The _SubscriptScriptLauncher tracks spawned child processes in self.procs and refuses to spawn a second batch: distributed training scripts are launched exactly once per launcher instance. Calling the launcher's launch path again after children were already created raises this RuntimeError to prevent duplicated process trees.

Solutions

1. Create a new Fabric/Trainer (and strategy/launcher) instance for each launch instead of reusing the old one
2. Restructure the script so the distributed run happens exactly once (e.g. one fit call, or sequential runs via fresh Trainers)
3. If you need repeated runs, move the loop inside the launched function rather than around the launcher

Example fix

# before
fabric = Fabric(strategy="ddp", devices=2)
fabric.run(train)
fabric.run(train)  # RuntimeError: launcher can only create subprocesses once

# after
for cfg in configs:
    fabric = Fabric(strategy="ddp", devices=2)  # fresh launcher each run
    fabric.run(partial(train, cfg=cfg))

if getattr(launcher, "procs", None):
    launcher = type(launcher)(strategy)  # fresh launcher for a new run

When it happens

Trigger: Invoking trainer/launcher .launch() or run() twice with the same subprocess-script based strategy (e.g. 'ddp') within one process, or reusing a Fabric/Trainer instance whose launcher already spawned procs.

Common situations: Running fit() then a second fit()/validate()/predict() that re-enters the launching code with the same strategy object; loops that rerun training in one script; re-invoking after catching an exception without recreating the strategy.

Related errors

• The launcher can only create subprocesses once.
• Accessing the device mesh before processes have initialized…
• Blocking backward sync is only possible if the module…
• It is recommended to use `self.log
• Lightning attempted to launch new distributed processes…

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/2135d4a71dd54552. Report an issue: GitHub.

Appendix: source

Thrown at src/lightning/fabric/strategies/launchers/subprocess_script.py:149

            # start process
            # if hydra is available and initialized, make sure to set the cwd correctly
            hydra_in_use = False
            cwd: Optional[str] = None
            if _HYDRA_AVAILABLE:
                from hydra.core.hydra_config import HydraConfig

                hydra_in_use = HydraConfig.initialized()
            if hydra_in_use:
                command, cwd = _hydra_subprocess_cmd(local_rank=local_rank)
            else:
                command = _basic_subprocess_cmd()

            proc = subprocess.Popen(command, env=env_copy, cwd=cwd)
            self.procs.append(proc)

    def _check_can_spawn_children(self) -> None:
        if len(self.procs) > 0:
            raise RuntimeError("The launcher can only create subprocesses once.")
        if self.cluster_environment.local_rank() != 0:
            raise RuntimeError(
                "Lightning attempted to launch new distributed processes with `local_rank > 0`. This should not happen."
                " Possible reasons: 1) LOCAL_RANK environment variable was incorrectly modified by the user,"
                " 2) `ClusterEnvironment.creates_processes_externally` incorrectly implemented."
            )


def _basic_subprocess_cmd() -> Sequence[str]:
    import __main__  # local import to avoid https://github.com/Lightning-AI/pytorch-lightning/issues/15218

    if __main__.__spec__ is None:  # pragma: no-cover
        return [sys.executable, os.path.abspath(sys.argv[0])] + sys.argv[1:]
    return [sys.executable, "-m", __main__.__spec__.name] + sys.argv[1:]


def _hydra_subprocess_cmd(local_rank: int) -> tuple[Sequence[str], str]:
    from hydra.core.hydra_config import HydraConfig

View on GitHub (pinned to 9fed5c27d2)