Lightning-AI/pytorch-lightning · error · ValueError

Did not find the environment variable `LSB_DJOB_RANKFILE`

Error message

Did not find the environment variable `LSB_DJOB_RANKFILE`

What it means

`LSFEnvironment._read_hosts` reads `LSB_DJOB_RANKFILE` (set by LSF's bsub/jsrun to a file listing host allocation order) to compute node rank and the main address. If the variable is not present in the environment, it raises this ValueError.

Solutions

  1. Submit the job through bsub/jsrun so LSF exports `LSB_DJOB_RANKFILE`: `bsub ... jsrun ... python train.py`
  2. For non-LSF execution, switch to a matching cluster environment instead of LSFEnvironment
  3. If manually emulating, export the variable pointing to a host file (each line a hostname)

Example fix

# before
python train.py  # LSB_DJOB_RANKFILE unset

# after
bsub -Is jsrun -n 4 python train.py  # LSF exports LSB_DJOB_RANKFILE
Defensive patterns

Strategy: validation

Validate before calling

import os
assert os.environ.get('LSB_DJOB_RANKFILE'), 'submit via bsub/jsrun so LSF exports LSB_DJOB_RANKFILE'

Prevention

When it happens

Trigger: Running with LSFEnvironment where `_get_node_rank` or `_get_main_address` is called but `LSB_DJOB_RANKFILE` is unset — process not submitted through bsub/jsrun, or a job context where LSF did not export the rankfile variable.

Common situations: Manual/interactive runs on LSF clusters outside bsub; scripts in containers that drop LSF env vars; porting LSF launch scripts between clusters where variable export differs.

Understand the failure class

Background: "environment variable is not set" and "Missing keys in environment" errors: what missing required env var messages mean and how to fix them — this error's family across 28 libraries.

Related errors

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/d85ff6210655d5d5. Report an issue: GitHub.

Appendix: source

Thrown at src/lightning/fabric/plugins/environments/lsf.py:164

        count: dict[str, int] = {}
        for host in hosts:
            if host not in count:
                count[host] = len(count)
        return count[socket.gethostname()]

    @staticmethod
    def _read_hosts() -> list[str]:
        """Read compute hosts that are a part of the compute job.

        LSF uses the Job Step Manager (JSM) to manage job steps. Job steps are executed by the JSM from "launch" nodes.
        Each job is assigned a launch node. This launch node will be the first node in the list contained in
        ``LSB_DJOB_RANKFILE``.

        """
        var = "LSB_DJOB_RANKFILE"
        rankfile = os.environ.get(var)
        if rankfile is None:
            raise ValueError("Did not find the environment variable `LSB_DJOB_RANKFILE`")
        if not rankfile:
            raise ValueError("The environment variable `LSB_DJOB_RANKFILE` is empty")

        fs = get_filesystem(rankfile)
        with fs.open(rankfile, "r") as f:
            ret = [line.strip() for line in f]
        # remove the launch node (i.e. the first node in LSB_DJOB_RANKFILE) from the list
        return ret[1:]

    def _get_main_address(self) -> str:
        """A helper for getting the main address.

        The main address is assigned to the first node in the list of nodes used for the job.

        """
        hosts = self._read_hosts()
        return hosts[0]

View on GitHub (pinned to 9fed5c27d2)