Lightning-AI/pytorch-lightning · error · ValueError

Cannot determine global rank. Environment variable `JSM_NAME

Error message

Cannot determine global rank. Environment variable `JSM_NAMESPACE_RANK` not found. Make sure you run your executable with `jsrun`.

What it means

`LSFEnvironment.global_rank()` reads `JSM_NAMESPACE_RANK`, exported only when the process is launched with jsrun under IBM Spectrum Launch. Missing variable means the code is not running inside a jsrun-launched LSF job.

Source

Thrown at src/lightning/fabric/plugins/environments/lsf.py:111

        """The world size is read from the environment variable ``JSM_NAMESPACE_SIZE``."""
        world_size = os.environ.get("JSM_NAMESPACE_SIZE")
        if world_size is None:
            raise ValueError(
                "Cannot determine world size. Environment variable `JSM_NAMESPACE_SIZE` not found."
                " Make sure you run your executable with `jsrun`."
            )
        return int(world_size)

    @override
    def set_world_size(self, size: int) -> None:
        log.debug("LSFEnvironment.set_world_size was called, but setting world size is not allowed. Ignored.")

    @override
    def global_rank(self) -> int:
        """The world size is read from the environment variable ``JSM_NAMESPACE_RANK``."""
        global_rank = os.environ.get("JSM_NAMESPACE_RANK")
        if global_rank is None:
            raise ValueError(
                "Cannot determine global rank. Environment variable `JSM_NAMESPACE_RANK` not found."
                " Make sure you run your executable with `jsrun`."
            )
        return int(global_rank)

    @override
    def set_global_rank(self, rank: int) -> None:
        log.debug("LSFEnvironment.set_global_rank was called, but setting global rank is not allowed. Ignored.")

    @override
    def local_rank(self) -> int:
        """The local rank is read from the environment variable `JSM_NAMESPACE_LOCAL_RANK`."""
        local_rank = os.environ.get("JSM_NAMESPACE_LOCAL_RANK")
        if local_rank is None:
            raise ValueError(
                "Cannot determine local rank. Environment variable `JSM_NAMESPACE_LOCAL_RANK` not found."
                " Make sure you run your executable with `jsrun`."
            )

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Run via jsrun so JSM_NAMESPACE_RANK is set: `jsrun ... python train.py`
2. If not on LSF, configure a different cluster environment matching your launcher
3. Sanity-check env inside the job: `echo $JSM_NAMESPACE_RANK`

Example fix

# before
python train.py  # no JSM_NAMESPACE_RANK

# after
jsrun -n 4 -a 1 python train.py  # sets JSM_NAMESPACE_RANK per process

import os
assert os.environ.get('JSM_NAMESPACE_RANK') is not None, 'not running under jsrun'

When it happens

Trigger: Calling `global_rank` (directly or through Fabric/Trainer startup with LSFEnvironment) when `JSM_NAMESPACE_RANK` is unset — plain python launch, wrong scheduler, or an env-stripping wrapper.

Common situations: Same as other LSF errors: running outside jsrun on LSF clusters, wrapper scripts that don't propagate the environment, debugging rank logic locally.

Understand the failure class

Background: "environment variable is not set" and "Missing keys in environment" errors: what missing required env var messages mean and how to fix them — this error's family across 28 libraries.

Related errors

• Cannot determine world size. Environment variable `JSM_NAMES
• Cannot determine local rank. Environment variable `JSM_NAMES
• Did not find the environment variable `LSB_DJOB_RANKFILE`
• `num_nodes` must be a positive integer, but got {num_nodes}.
• To use Fabric with more than one device, you must call `.lau

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/f9441bcca3c6c7c0. Report an issue: GitHub.