Lightning-AI/pytorch-lightning · error · ValueError

Unable to find 'latest' file at {latest_path}

Error message

Unable to find 'latest' file at {latest_path}

What it means

Raised as a ValueError by Lightning's ds_checkpoint_dir helper when converting a DeepSpeed ZeRO checkpoint: no tag was given and the 'latest' file, which DeepSpeed writes into the checkpoint directory to record the most recent tag, was not found. Without a tag, the location of the sharded ZeRO files cannot be determined.

Source

Thrown at src/lightning/pytorch/utilities/deepspeed.py:36

import os
from typing import Any

import torch

from lightning.fabric.utilities.types import _PATH
from lightning.pytorch.strategies.deepspeed import _DEEPSPEED_AVAILABLE

CPU_DEVICE = torch.device("cpu")


def ds_checkpoint_dir(checkpoint_dir: _PATH, tag: str | None = None) -> str:
    if tag is None:
        latest_path = os.path.join(checkpoint_dir, "latest")
        if os.path.isfile(latest_path):
            with open(latest_path) as fd:
                tag = fd.read().strip()
        else:
            raise ValueError(f"Unable to find 'latest' file at {latest_path}")

    directory = os.path.join(checkpoint_dir, tag)

    if not os.path.isdir(directory):
        raise FileNotFoundError(f"Directory '{ds_checkpoint_dir}' doesn't exist")
    return directory


# Modified script from https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/utils/zero_to_fp32.py
def convert_zero_checkpoint_to_fp32_state_dict(
    checkpoint_dir: _PATH, output_file: _PATH, tag: str | None = None
) -> dict[str, Any]:
    """Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict`` file that can be loaded with
    ``torch.load(file)`` + ``load_state_dict()`` and used for training without DeepSpeed. It gets copied into the top
    level checkpoint dir, so the user can easily do the conversion at any point in the future. Once extracted, the
    weights don't require DeepSpeed and can be used in any application. Additionally the script has been modified to
    ensure we keep the lightning state inside the state dict for being able to run
    ``LightningModule.load_from_checkpoint('...')```.

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Pass the tag explicitly: convert_zero_checkpoint_to_fp32_state_dict(ckpt_dir, out_file, tag='<existing-tag-folder>')
2. List os.listdir(ckpt_dir) and use the tag subdirectory name that exists
3. Re-save the checkpoint from DeepSpeed/Lightning so the 'latest' file is written

Example fix

# before
convert_zero_checkpoint_to_fp32_state_dict('checkpoints/last', 'model.pt')

# after
convert_zero_checkpoint_to_fp32_state_dict('checkpoints/last', 'model.pt', tag='2000')

import os
ckpt_dir = 'path/to/ckpt'
if not os.path.isfile(os.path.join(ckpt_dir, 'latest')):
    tags = sorted(d for d in os.listdir(ckpt_dir) if d.isdigit())
    assert tags, f'no tag dirs in {ckpt_dir}'
    tag = tags[-1]
else:
    tag = None
convert_zero_checkpoint_to_fp32_state_dict(ckpt_dir, 'model.pt', tag=tag)

try:
    convert_zero_checkpoint_to_fp32_state_dict(ckpt_dir, out)
except ValueError as e:
    if "Unable to find 'latest'" in str(e):
        tag = sorted(d for d in os.listdir(ckpt_dir) if d.isdigit())[-1]
        convert_zero_checkpoint_to_fp32_state_dict(ckpt_dir, out, tag=tag)
    else:
        raise

When it happens

Trigger: Calling convert_zero_checkpoint_to_fp32_state_dict(checkpoint_dir, output_file) without tag= on a directory that lacks a 'latest' file; common with checkpoints saved by DeepSpeed versions/paths that only write numbered tag subdirectories, or when pointing at a copied/partial checkpoint directory.

Common situations: Checkpoint dir was copied without the 'latest' marker; ZeRO stage 2/3 checkpoints where the tag folder exists but the marker is missing; manually assembled checkpoint folders.

Related errors

• Directory '{ds_checkpoint_dir}' doesn't exist
• `Trainer.save_checkpoint(..., storage_options=...)` with `st
• `precision={precision!r})` is not supported in DeepSpeed. `p
• To use the `DeepSpeedStrategy`, you must have DeepSpeed inst
• PyTorch >= 2.6 requires DeepSpeed >= 0.16.0. Detected DeepSp

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/5a9fe2782db70f29. Report an issue: GitHub.