Lightning-AI/pytorch-lightning · error · ValueError

Cannot determine local rank. Environment variable `JSM_NAMES

Error message

Cannot determine local rank. Environment variable `JSM_NAMESPACE_LOCAL_RANK` not found. Make sure you run your executable with `jsrun`.

What it means

`LSFEnvironment.local_rank()` reads `JSM_NAMESPACE_LOCAL_RANK`, set per-process by jsrun on LSF. Without it (not launched by jsrun), reading local_rank raises ValueError.

Source

Thrown at src/lightning/fabric/plugins/environments/lsf.py:126

        """The world size is read from the environment variable ``JSM_NAMESPACE_RANK``."""
        global_rank = os.environ.get("JSM_NAMESPACE_RANK")
        if global_rank is None:
            raise ValueError(
                "Cannot determine global rank. Environment variable `JSM_NAMESPACE_RANK` not found."
                " Make sure you run your executable with `jsrun`."
            )
        return int(global_rank)

    @override
    def set_global_rank(self, rank: int) -> None:
        log.debug("LSFEnvironment.set_global_rank was called, but setting global rank is not allowed. Ignored.")

    @override
    def local_rank(self) -> int:
        """The local rank is read from the environment variable `JSM_NAMESPACE_LOCAL_RANK`."""
        local_rank = os.environ.get("JSM_NAMESPACE_LOCAL_RANK")
        if local_rank is None:
            raise ValueError(
                "Cannot determine local rank. Environment variable `JSM_NAMESPACE_LOCAL_RANK` not found."
                " Make sure you run your executable with `jsrun`."
            )
        return int(local_rank)

    @override
    def node_rank(self) -> int:
        """The node rank is determined by the position of the current hostname in the OpenMPI host rank file stored in
        ``LSB_DJOB_RANKFILE``."""
        return self._node_rank

    def _get_node_rank(self) -> int:
        """A helper method for getting the node rank.

        The node rank is determined by the position of the current node in the list of hosts used in the job. This is
        calculated by reading all hosts from ``LSB_DJOB_RANKFILE`` and finding this node's hostname in the list.

        """

View on GitHub (pinned to 9fed5c27d2)

Solutions

Launch with jsrun (`jsrun ... python train.py`) so JSM_NAMESPACE_LOCAL_RANK is exported
For local debugging, temporarily set `JSM_NAMESPACE_LOCAL_RANK=0` (and the sibling vars) or use a non-LSF environment
Verify the variable inside the job before running heavy work

Example fix

# before
python train.py  # JSM_NAMESPACE_LOCAL_RANK missing

# after
JSM_NAMESPACE_LOCAL_RANK=0 JSM_NAMESPACE_RANK=0 JSM_NAMESPACE_SIZE=1 python train.py  # local debug
# or properly: jsrun -n 1 python train.py

Defensive patterns

Strategy: validation

Validate before calling

import os
if os.environ.get('JSM_NAMESPACE_LOCAL_RANK') is None:
    os.environ.setdefault('JSM_NAMESPACE_LOCAL_RANK', '0')  # local debug only

Prevention

Launch with jsrun; for local debugging set JSM_NAMESPACE_* stubs
Verify env inside the allocated job before training

When it happens

Trigger: Accessing `local_rank` with LSFEnvironment outside a jsrun launch — e.g. `python train.py` on a front-end node, or a job script where the variable name changed/was cleared.

Common situations: Adapting multi-node LSF scripts; local debugging of device placement code that queries local_rank; env var stripped by container runtimes inside jsrun.

Understand the failure class

Background: "environment variable is not set" and "Missing keys in environment" errors: what missing required env var messages mean and how to fix them — this error's family across 28 libraries.

Related errors

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/dc96155ed261cb5e. Report an issue: GitHub.