Lightning-AI/pytorch-lightning · error · FileNotFoundError

Directory '{ds_checkpoint_dir}' doesn't exist

Error message

Directory '{ds_checkpoint_dir}' doesn't exist

What it means

Raised as FileNotFoundError by ds_checkpoint_dir when the resolved checkpoint tag directory (checkpoint_dir joined with the tag from the 'latest' file or the tag argument) does not exist on disk. The tag was obtained, but the folder holding the DeepSpeed sharded files is missing, so conversion cannot proceed.

Source

Thrown at src/lightning/pytorch/utilities/deepspeed.py:41

from lightning.fabric.utilities.types import _PATH
from lightning.pytorch.strategies.deepspeed import _DEEPSPEED_AVAILABLE

CPU_DEVICE = torch.device("cpu")


def ds_checkpoint_dir(checkpoint_dir: _PATH, tag: str | None = None) -> str:
    if tag is None:
        latest_path = os.path.join(checkpoint_dir, "latest")
        if os.path.isfile(latest_path):
            with open(latest_path) as fd:
                tag = fd.read().strip()
        else:
            raise ValueError(f"Unable to find 'latest' file at {latest_path}")

    directory = os.path.join(checkpoint_dir, tag)

    if not os.path.isdir(directory):
        raise FileNotFoundError(f"Directory '{ds_checkpoint_dir}' doesn't exist")
    return directory


# Modified script from https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/utils/zero_to_fp32.py
def convert_zero_checkpoint_to_fp32_state_dict(
    checkpoint_dir: _PATH, output_file: _PATH, tag: str | None = None
) -> dict[str, Any]:
    """Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict`` file that can be loaded with
    ``torch.load(file)`` + ``load_state_dict()`` and used for training without DeepSpeed. It gets copied into the top
    level checkpoint dir, so the user can easily do the conversion at any point in the future. Once extracted, the
    weights don't require DeepSpeed and can be used in any application. Additionally the script has been modified to
    ensure we keep the lightning state inside the state dict for being able to run
    ``LightningModule.load_from_checkpoint('...')```.

    Args:
        checkpoint_dir: path to the desired checkpoint folder.
            (one that contains the tag-folder, like ``global_step14``)
        output_file: path to the pytorch fp32 state_dict output file (e.g. path/pytorch_model.bin)

View on GitHub (pinned to 9fed5c27d2)

Solutions

  1. Verify the tag directory exists: ls <checkpoint_dir>/<tag> and correct the tag argument
  2. If 'latest' is stale, edit it to an existing tag or pass tag= explicitly
  3. Confirm checkpoint_dir is an absolute path (avoid CWD-relative mismatches)

Example fix

# before
convert_zero_checkpoint_to_fp32_state_dict('ckpt', 'model.pt', tag='1000')  # ckpt/1000 missing

# after
import os
tags = [d for d in os.listdir('ckpt') if d.isdigit()]
convert_zero_checkpoint_to_fp32_state_dict('ckpt', 'model.pt', tag=tags[-1])
Defensive patterns

Strategy: validation

Validate before calling

import os
ckpt_dir, tag = 'path/to/ckpt', '2000'
if tag is None:
    with open(os.path.join(ckpt_dir, 'latest')) as f:
        tag = f.read().strip()
assert os.path.isdir(os.path.join(ckpt_dir, tag)), f'tag dir {tag} missing in {ckpt_dir}'

Try / catch

from lightning.pytorch.utilities.deepspeed import convert_zero_checkpoint_to_fp32_state_dict
try:
    convert_zero_checkpoint_to_fp32_state_dict(ckpt_dir, out, tag=tag)
except FileNotFoundError:
    tag = sorted(d for d in os.listdir(ckpt_dir) if d.isdigit())[-1]
    convert_zero_checkpoint_to_fp32_state_dict(ckpt_dir, out, tag=tag)

Prevention

When it happens

Trigger: convert_zero_checkpoint_to_fp32_state_dict with a tag whose subdirectory isn't present, or a 'latest' file containing a stale tag; also relative-path issues where the working directory differs from the training run.

Common situations: 'latest' points to a tag that was deleted or never synced; checkpoint directory moved between machines; typo in the tag argument; partial uploads of distributed checkpoints.

Related errors

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/bccd255d89b36ea2. Report an issue: GitHub.