{"record":{"id":"dc96155ed261cb5e","repo":"Lightning-AI/pytorch-lightning","slug":"cannot-determine-local-rank-environment-variable","errorCode":null,"errorMessage":"Cannot determine local rank. Environment variable `JSM_NAMESPACE_LOCAL_RANK` not found. Make sure you run your executable with `jsrun`.","messagePattern":"Cannot determine local rank\\. Environment variable `JSM_NAMESPACE_LOCAL_RANK` not found\\. Make sure you run your executable with `jsrun`\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/lightning/fabric/plugins/environments/lsf.py","lineNumber":126,"sourceCode":"        \"\"\"The world size is read from the environment variable ``JSM_NAMESPACE_RANK``.\"\"\"\n        global_rank = os.environ.get(\"JSM_NAMESPACE_RANK\")\n        if global_rank is None:\n            raise ValueError(\n                \"Cannot determine global rank. Environment variable `JSM_NAMESPACE_RANK` not found.\"\n                \" Make sure you run your executable with `jsrun`.\"\n            )\n        return int(global_rank)\n\n    @override\n    def set_global_rank(self, rank: int) -> None:\n        log.debug(\"LSFEnvironment.set_global_rank was called, but setting global rank is not allowed. Ignored.\")\n\n    @override\n    def local_rank(self) -> int:\n        \"\"\"The local rank is read from the environment variable `JSM_NAMESPACE_LOCAL_RANK`.\"\"\"\n        local_rank = os.environ.get(\"JSM_NAMESPACE_LOCAL_RANK\")\n        if local_rank is None:\n            raise ValueError(\n                \"Cannot determine local rank. Environment variable `JSM_NAMESPACE_LOCAL_RANK` not found.\"\n                \" Make sure you run your executable with `jsrun`.\"\n            )\n        return int(local_rank)\n\n    @override\n    def node_rank(self) -> int:\n        \"\"\"The node rank is determined by the position of the current hostname in the OpenMPI host rank file stored in\n        ``LSB_DJOB_RANKFILE``.\"\"\"\n        return self._node_rank\n\n    def _get_node_rank(self) -> int:\n        \"\"\"A helper method for getting the node rank.\n\n        The node rank is determined by the position of the current node in the list of hosts used in the job. This is\n        calculated by reading all hosts from ``LSB_DJOB_RANKFILE`` and finding this node's hostname in the list.\n\n        \"\"\"","sourceCodeStart":108,"sourceCodeEnd":144,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/fabric/plugins/environments/lsf.py#L108-L144","documentation":"`LSFEnvironment.local_rank()` reads `JSM_NAMESPACE_LOCAL_RANK`, set per-process by jsrun on LSF. Without it (not launched by jsrun), reading local_rank raises ValueError.","triggerScenarios":"Accessing `local_rank` with LSFEnvironment outside a jsrun launch — e.g. `python train.py` on a front-end node, or a job script where the variable name changed/was cleared.","commonSituations":"Adapting multi-node LSF scripts; local debugging of device placement code that queries local_rank; env var stripped by container runtimes inside jsrun.","solutions":["Launch with jsrun (`jsrun ... python train.py`) so JSM_NAMESPACE_LOCAL_RANK is exported","For local debugging, temporarily set `JSM_NAMESPACE_LOCAL_RANK=0` (and the sibling vars) or use a non-LSF environment","Verify the variable inside the job before running heavy work"],"exampleFix":"# before\npython train.py  # JSM_NAMESPACE_LOCAL_RANK missing\n\n# after\nJSM_NAMESPACE_LOCAL_RANK=0 JSM_NAMESPACE_RANK=0 JSM_NAMESPACE_SIZE=1 python train.py  # local debug\n# or properly: jsrun -n 1 python train.py","handlingStrategy":"validation","validationCode":"import os\nif os.environ.get('JSM_NAMESPACE_LOCAL_RANK') is None:\n    os.environ.setdefault('JSM_NAMESPACE_LOCAL_RANK', '0')  # local debug only","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Launch with jsrun; for local debugging set JSM_NAMESPACE_* stubs","Verify env inside the allocated job before training"],"tags":["lsf","jsrun","hpc","missing-env-var","distributed"],"backgroundTag":"missing-env-var","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}