{"record":{"id":"d8d2bbd2c1141082","repo":"Lightning-AI/pytorch-lightning","slug":"lightning-attempted-to-launch-new-distributed-proc","errorCode":null,"errorMessage":"Lightning attempted to launch new distributed processes with `local_rank > 0`. This should not happen. Possible reasons: 1) LOCAL_RANK environment variable was incorrectly modified by the user, 2) `ClusterEnvironment.creates_processes_externally` incorrectly implemented.","messagePattern":"Lightning attempted to launch new distributed processes with `local_rank > 0`\\. This should not happen\\. Possible reasons: 1\\) LOCAL_RANK environment variable was incorrectly modified by the user, 2\\) `ClusterEnvironment\\.creates_processes_externally` incorrectly implemented\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"src/lightning/fabric/strategies/launchers/subprocess_script.py","lineNumber":151,"sourceCode":"            hydra_in_use = False\n            cwd: Optional[str] = None\n            if _HYDRA_AVAILABLE:\n                from hydra.core.hydra_config import HydraConfig\n\n                hydra_in_use = HydraConfig.initialized()\n            if hydra_in_use:\n                command, cwd = _hydra_subprocess_cmd(local_rank=local_rank)\n            else:\n                command = _basic_subprocess_cmd()\n\n            proc = subprocess.Popen(command, env=env_copy, cwd=cwd)\n            self.procs.append(proc)\n\n    def _check_can_spawn_children(self) -> None:\n        if len(self.procs) > 0:\n            raise RuntimeError(\"The launcher can only create subprocesses once.\")\n        if self.cluster_environment.local_rank() != 0:\n            raise RuntimeError(\n                \"Lightning attempted to launch new distributed processes with `local_rank > 0`. This should not happen.\"\n                \" Possible reasons: 1) LOCAL_RANK environment variable was incorrectly modified by the user,\"\n                \" 2) `ClusterEnvironment.creates_processes_externally` incorrectly implemented.\"\n            )\n\n\ndef _basic_subprocess_cmd() -> Sequence[str]:\n    import __main__  # local import to avoid https://github.com/Lightning-AI/pytorch-lightning/issues/15218\n\n    if __main__.__spec__ is None:  # pragma: no-cover\n        return [sys.executable, os.path.abspath(sys.argv[0])] + sys.argv[1:]\n    return [sys.executable, \"-m\", __main__.__spec__.name] + sys.argv[1:]\n\n\ndef _hydra_subprocess_cmd(local_rank: int) -> tuple[Sequence[str], str]:\n    from hydra.core.hydra_config import HydraConfig\n    from hydra.utils import get_original_cwd, to_absolute_path\n","sourceCodeStart":133,"sourceCodeEnd":169,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/fabric/strategies/launchers/subprocess_script.py#L133-L169","documentation":"The subprocess-script launcher should only ever spawn children from the rank-0 process. If cluster_environment.local_rank() returns something other than 0 at spawn time, either the LOCAL_RANK env var was tampered with or the ClusterEnvironment reports creates_processes_externally incorrectly, and Lightning aborts to avoid every rank spawning a full copy of the job.","triggerScenarios":"Manually setting/exporting LOCAL_RANK (or PL_GLOBAL_RANK-related vars) before running the script; using a custom ClusterEnvironment whose creates_processes_externally/local_rank are inconsistent; double-launching under an existing torchrun session.","commonSituations":"Running 'LOCAL_RANK=2 python train.py' by mistake; a wrapper script or scheduler injecting LOCAL_RANK; copying a launch command from a torchrun context into a plain python invocation; custom cluster plugins with wrong external-process flags.","solutions":["Unset LOCAL_RANK ('unset LOCAL_RANK' / remove it from env) and launch with plain python or fabric's CLI so Lightning spawns ranks itself","If you intend torchrun/srun to create processes, use the cluster environment/strategy designed for external launching (e.g. 'ddp' with creates_processes_externally=True)","Audit custom ClusterEnvironment implementations so local_rank() and creates_processes_externally agree"],"exampleFix":"# before\nLOCAL_RANK=1 python train.py --strategy ddp\n\n# after\nunset LOCAL_RANK\npython train.py --strategy ddp  # Lightning spawns all ranks from rank 0","handlingStrategy":"validation","validationCode":"import os\nif int(os.environ.get(\"LOCAL_RANK\", \"0\")) != 0 and launcher_is_subprocess_script:\n    raise RuntimeError(\"LOCAL_RANK must be unset/0 for Lightning-spawned runs\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Don't set LOCAL_RANK manually; let torchrun or Lightning own it","Sanitize the environment in wrapper scripts that may inherit rank variables"],"tags":["pytorch-lightning","local-rank","environment-variables","distributed","launcher"],"backgroundTag":"env-variable-rank-misconfiguration","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}