Lightning-AI/pytorch-lightning · error · ValueError

Cannot determine world size. Environment variable `JSM_NAMES

Error message

Cannot determine world size. Environment variable `JSM_NAMESPACE_SIZE` not found. Make sure you run your executable with `jsrun`.

What it means

`LSFEnvironment.world_size()` reads `JSM_NAMESPACE_SIZE`, set by IBM Spectrum Launch (jsrun) on LSF clusters. If the variable is absent — the process was not launched via `jsrun` — it raises ValueError telling you to run your executable with jsrun.

Source

Thrown at src/lightning/fabric/plugins/environments/lsf.py:96

    @property
    @override
    def main_port(self) -> int:
        """The main port is calculated from the LSF job ID."""
        return self._main_port

    @staticmethod
    @override
    def detect() -> bool:
        """Returns ``True`` if the current process was launched using the ``jsrun`` command."""
        required_env_vars = {"LSB_JOBID", "LSB_DJOB_RANKFILE", "JSM_NAMESPACE_LOCAL_RANK", "JSM_NAMESPACE_SIZE"}
        return required_env_vars.issubset(os.environ.keys())

    @override
    def world_size(self) -> int:
        """The world size is read from the environment variable ``JSM_NAMESPACE_SIZE``."""
        world_size = os.environ.get("JSM_NAMESPACE_SIZE")
        if world_size is None:
            raise ValueError(
                "Cannot determine world size. Environment variable `JSM_NAMESPACE_SIZE` not found."
                " Make sure you run your executable with `jsrun`."
            )
        return int(world_size)

    @override
    def set_world_size(self, size: int) -> None:
        log.debug("LSFEnvironment.set_world_size was called, but setting world size is not allowed. Ignored.")

    @override
    def global_rank(self) -> int:
        """The world size is read from the environment variable ``JSM_NAMESPACE_RANK``."""
        global_rank = os.environ.get("JSM_NAMESPACE_RANK")
        if global_rank is None:
            raise ValueError(
                "Cannot determine global rank. Environment variable `JSM_NAMESPACE_RANK` not found."
                " Make sure you run your executable with `jsrun`."
            )

View on GitHub (pinned to 9fed5c27d2)

Solutions

  1. Launch through jsrun: `jsrun -n <ranks> -a <gpu_per_rank> -c <cpus> python train.py`
  2. If you didn't intend LSF, use the correct environment (e.g. LightningEnvironment/manual) for your launcher
  3. Verify with `echo $JSM_NAMESPACE_SIZE` inside the same job context

Example fix

# before (login node / plain python)
python train.py  # JSM_NAMESPACE_SIZE unset -> ValueError

# after
jsrun -n 4 -a 1 -c 6 -g 1 python train.py
Defensive patterns

Strategy: validation

Validate before calling

import os
assert os.environ.get('JSM_NAMESPACE_SIZE'), 'launch with jsrun (JSM_NAMESPACE_SIZE unset)'

Prevention

When it happens

Trigger: Accessing `world_size` (directly or via strategy startup) with `LSFEnvironment` while `JSM_NAMESPACE_SIZE` is unset: e.g. running `python train.py` directly on a login node instead of `jsrun --np ... train.py`, or a mis-set-up jsrun invocation.

Common situations: IBM LSF/HPC clusters (Summit-class systems); debugging scripts outside the scheduler; sbatch/jsrun wrappers that drop environment variables; typos in jsrun launch commands.

Understand the failure class

Background: "environment variable is not set" and "Missing keys in environment" errors: what missing required env var messages mean and how to fix them — this error's family across 28 libraries.

Related errors

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/e72c263799dfd68f. Report an issue: GitHub.