{"record":{"id":"df4ce323e297884a","repo":"Lightning-AI/pytorch-lightning","slug":"lightning-attempted-to-launch-new-distributed-proc-df4ce3","errorCode":null,"errorMessage":"Lightning attempted to launch new distributed processes with `local_rank > 0`. This should not happen. Possible reasons: 1) LOCAL_RANK environment variable was incorrectly modified by the user, 2) `ClusterEnvironment.creates_processes_externally` incorrectly implemented.","messagePattern":"Lightning attempted to launch new distributed processes with `local_rank > 0`\\. This should not happen\\. Possible reasons: 1\\) LOCAL_RANK environment variable was incorrectly modified by the user, 2\\) `ClusterEnvironment\\.creates_processes_externally` incorrectly implemented\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"src/lightning/pytorch/strategies/launchers/subprocess_script.py","lineNumber":155,"sourceCode":"            cwd: Optional[str] = None\n            if _HYDRA_AVAILABLE:\n                from hydra.core.hydra_config import HydraConfig\n\n                hydra_in_use = HydraConfig.initialized()\n\n            if hydra_in_use:\n                command, cwd = _hydra_subprocess_cmd(local_rank)\n            else:\n                command = _basic_subprocess_cmd()\n\n            new_process = subprocess.Popen(command, env=env_copy, cwd=cwd)\n            self.procs.append(new_process)\n\n    def _check_can_spawn_children(self) -> None:\n        if len(self.procs) > 0:\n            raise RuntimeError(\"The launcher can only create subprocesses once.\")\n        if self.cluster_environment.local_rank() != 0:\n            raise RuntimeError(\n                \"Lightning attempted to launch new distributed processes with `local_rank > 0`. This should not happen.\"\n                \" Possible reasons: 1) LOCAL_RANK environment variable was incorrectly modified by the user,\"\n                \" 2) `ClusterEnvironment.creates_processes_externally` incorrectly implemented.\"\n            )\n","sourceCodeStart":137,"sourceCodeEnd":160,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/pytorch/strategies/launchers/subprocess_script.py#L137-L160","documentation":"_SubprocessScriptLauncher is only supposed to spawn worker processes from the process with local rank 0. If cluster_environment.local_rank() != 0 at spawn time, something is wrong: the LOCAL_RANK env var was tampered with, or a custom ClusterEnvironment implementing creates_processes_externally incorrectly let every rank try to launch.","triggerScenarios":"Setting/modifying the LOCAL_RANK environment variable manually before running the script; using a custom ClusterEnvironment whose creates_processes_externally returns False while ranks other than 0 reach _check_can_spawn_children; launching via a tool that exports LOCAL_RANK>0 (e.g. torchrun leftover env) while using a strategy that spawns its own processes.","commonSituations":"Switching a script between torchrun (external spawning) and Lightning-internal spawning without clearing LOCAL_RANK/related env vars; buggy custom cluster environments in house-made plugins.","solutions":["Unset leftover launcher env vars (LOCAL_RANK, RANK, WORLD_SIZE, etc.) before running standalone: e.g. env -u LOCAL_RANK python train.py","If using torchrun/torchelastic, use the strategy/launcher designed for external process creation (creates_processes_externally=True, e.g. _SubprocessScriptLauncher via XLAStrategy/defaults)","Fix custom ClusterEnvironment.creates_processes_externally and local_rank() to report the truth"],"exampleFix":"# before\n# shell still has torchrun vars\nLOCAL_RANK=1 python train.py  # RuntimeError\n\n# after\nenv -u LOCAL_RANK -u RANK -u WORLD_SIZE python train.py","handlingStrategy":"validation","validationCode":"import os\nassert int(os.environ.get(\"LOCAL_RANK\", 0)) == 0, \"run from rank-0 process or unset LOCAL_RANK\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Clear LOCAL_RANK/RANK/WORLD_SIZE when running standalone after torchrun: env -u LOCAL_RANK ...","Use torchrun-compatible settings (creates_processes_externally) when an external launcher owns process creation","Test custom ClusterEnvironment.local_rank() against the real env"],"tags":["local-rank","distributed","subprocess","cluster-environment"],"backgroundTag":"local-rank-env-conflict","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}