{"record":{"id":"2135d4a71dd54552","repo":"Lightning-AI/pytorch-lightning","slug":"the-launcher-can-only-create-subprocesses-once","errorCode":null,"errorMessage":"The launcher can only create subprocesses once.","messagePattern":"The launcher can only create subprocesses once\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"src/lightning/fabric/strategies/launchers/subprocess_script.py","lineNumber":149,"sourceCode":"            # start process\n            # if hydra is available and initialized, make sure to set the cwd correctly\n            hydra_in_use = False\n            cwd: Optional[str] = None\n            if _HYDRA_AVAILABLE:\n                from hydra.core.hydra_config import HydraConfig\n\n                hydra_in_use = HydraConfig.initialized()\n            if hydra_in_use:\n                command, cwd = _hydra_subprocess_cmd(local_rank=local_rank)\n            else:\n                command = _basic_subprocess_cmd()\n\n            proc = subprocess.Popen(command, env=env_copy, cwd=cwd)\n            self.procs.append(proc)\n\n    def _check_can_spawn_children(self) -> None:\n        if len(self.procs) > 0:\n            raise RuntimeError(\"The launcher can only create subprocesses once.\")\n        if self.cluster_environment.local_rank() != 0:\n            raise RuntimeError(\n                \"Lightning attempted to launch new distributed processes with `local_rank > 0`. This should not happen.\"\n                \" Possible reasons: 1) LOCAL_RANK environment variable was incorrectly modified by the user,\"\n                \" 2) `ClusterEnvironment.creates_processes_externally` incorrectly implemented.\"\n            )\n\n\ndef _basic_subprocess_cmd() -> Sequence[str]:\n    import __main__  # local import to avoid https://github.com/Lightning-AI/pytorch-lightning/issues/15218\n\n    if __main__.__spec__ is None:  # pragma: no-cover\n        return [sys.executable, os.path.abspath(sys.argv[0])] + sys.argv[1:]\n    return [sys.executable, \"-m\", __main__.__spec__.name] + sys.argv[1:]\n\n\ndef _hydra_subprocess_cmd(local_rank: int) -> tuple[Sequence[str], str]:\n    from hydra.core.hydra_config import HydraConfig","sourceCodeStart":131,"sourceCodeEnd":167,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/fabric/strategies/launchers/subprocess_script.py#L131-L167","documentation":"The _SubscriptScriptLauncher tracks spawned child processes in self.procs and refuses to spawn a second batch: distributed training scripts are launched exactly once per launcher instance. Calling the launcher's launch path again after children were already created raises this RuntimeError to prevent duplicated process trees.","triggerScenarios":"Invoking trainer/launcher .launch() or run() twice with the same subprocess-script based strategy (e.g. 'ddp') within one process, or reusing a Fabric/Trainer instance whose launcher already spawned procs.","commonSituations":"Running fit() then a second fit()/validate()/predict() that re-enters the launching code with the same strategy object; loops that rerun training in one script; re-invoking after catching an exception without recreating the strategy.","solutions":["Create a new Fabric/Trainer (and strategy/launcher) instance for each launch instead of reusing the old one","Restructure the script so the distributed run happens exactly once (e.g. one fit call, or sequential runs via fresh Trainers)","If you need repeated runs, move the loop inside the launched function rather than around the launcher"],"exampleFix":"# before\nfabric = Fabric(strategy=\"ddp\", devices=2)\nfabric.run(train)\nfabric.run(train)  # RuntimeError: launcher can only create subprocesses once\n\n# after\nfor cfg in configs:\n    fabric = Fabric(strategy=\"ddp\", devices=2)  # fresh launcher each run\n    fabric.run(partial(train, cfg=cfg))","handlingStrategy":"fallback","validationCode":"if getattr(launcher, \"procs\", None):\n    launcher = type(launcher)(strategy)  # fresh launcher for a new run","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Treat Fabric/Trainer + subprocess launcher as single-use: recreate per run","Put hyperparameter loops inside the training function, not around the launcher"],"tags":["pytorch-lightning","subprocess","launcher","ddp","lifecycle"],"backgroundTag":"launcher-reuse-single-shot","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}