Lightning-AI/pytorch-lightning · error · ValueError

Did not find the environment variable `LSB_DJOB_RANKFILE`

Error message

Did not find the environment variable `LSB_DJOB_RANKFILE`

What it means

`LSFEnvironment._read_hosts` reads `LSB_DJOB_RANKFILE` (set by LSF's bsub/jsrun to a file listing host allocation order) to compute node rank and the main address. If the variable is not present in the environment, it raises this ValueError.

Source

Thrown at src/lightning/fabric/plugins/environments/lsf.py:164

        count: dict[str, int] = {}
        for host in hosts:
            if host not in count:
                count[host] = len(count)
        return count[socket.gethostname()]

    @staticmethod
    def _read_hosts() -> list[str]:
        """Read compute hosts that are a part of the compute job.

        LSF uses the Job Step Manager (JSM) to manage job steps. Job steps are executed by the JSM from "launch" nodes.
        Each job is assigned a launch node. This launch node will be the first node in the list contained in
        ``LSB_DJOB_RANKFILE``.

        """
        var = "LSB_DJOB_RANKFILE"
        rankfile = os.environ.get(var)
        if rankfile is None:
            raise ValueError("Did not find the environment variable `LSB_DJOB_RANKFILE`")
        if not rankfile:
            raise ValueError("The environment variable `LSB_DJOB_RANKFILE` is empty")

        fs = get_filesystem(rankfile)
        with fs.open(rankfile, "r") as f:
            ret = [line.strip() for line in f]
        # remove the launch node (i.e. the first node in LSB_DJOB_RANKFILE) from the list
        return ret[1:]

    def _get_main_address(self) -> str:
        """A helper for getting the main address.

        The main address is assigned to the first node in the list of nodes used for the job.

        """
        hosts = self._read_hosts()
        return hosts[0]

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Submit the job through bsub/jsrun so LSF exports `LSB_DJOB_RANKFILE`: `bsub ... jsrun ... python train.py`
2. For non-LSF execution, switch to a matching cluster environment instead of LSFEnvironment
3. If manually emulating, export the variable pointing to a host file (each line a hostname)

Example fix

# before
python train.py  # LSB_DJOB_RANKFILE unset

# after
bsub -Is jsrun -n 4 python train.py  # LSF exports LSB_DJOB_RANKFILE

import os
assert os.environ.get('LSB_DJOB_RANKFILE'), 'submit via bsub/jsrun so LSF exports LSB_DJOB_RANKFILE'

When it happens

Trigger: Running with LSFEnvironment where `_get_node_rank` or `_get_main_address` is called but `LSB_DJOB_RANKFILE` is unset — process not submitted through bsub/jsrun, or a job context where LSF did not export the rankfile variable.

Common situations: Manual/interactive runs on LSF clusters outside bsub; scripts in containers that drop LSF env vars; porting LSF launch scripts between clusters where variable export differs.

Understand the failure class

Background: "environment variable is not set" and "Missing keys in environment" errors: what missing required env var messages mean and how to fix them — this error's family across 28 libraries.

Related errors

• Cannot determine world size. Environment variable `JSM_NAMES
• Cannot determine global rank. Environment variable `JSM_NAME
• Cannot determine local rank. Environment variable `JSM_NAMES
• `num_nodes` must be a positive integer, but got {num_nodes}.
• To use Fabric with more than one device, you must call `.lau

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/d85ff6210655d5d5. Report an issue: GitHub.