Lightning-AI/pytorch-lightning · error · FileNotFoundError

The provided path is not a valid DeepSpeed checkpoint: {path

Error message

The provided path is not a valid DeepSpeed checkpoint: {path_str}

What it means

Fallback case of _validate_checkpoint_directory: the path is neither a DeepSpeed checkpoint, nor a subfolder/file of one. Lightning raises FileNotFoundError with the plain 'not a valid DeepSpeed checkpoint' message.

Source

Thrown at src/lightning/fabric/strategies/deepspeed.py:928

    path_is_ds_checkpoint = _is_deepspeed_checkpoint(path_str, fs)
    default_message = f"The provided path is not a valid DeepSpeed checkpoint: {path_str}"

    if not path_is_ds_checkpoint:
        # Case 1: User may have accidentally passed the subfolder "checkpoint"
        parent = os.path.dirname(path_str)
        if _is_deepspeed_checkpoint(parent, fs):
            raise FileNotFoundError(
                f"{default_message}. It looks like you passed the path to a subfolder."
                f" Try to load using this parent directory instead: {parent}"
            )
        # Case 2: User may have accidentally passed the path to a file inside the "checkpoint" subfolder
        grandparent = os.path.dirname(parent)
        if fs.isfile(path_str) and _is_deepspeed_checkpoint(grandparent, fs):
            raise FileNotFoundError(
                f"{default_message}. It looks like you passed the path to a file inside a DeepSpeed"
                f" checkpoint folder. Try to load using this parent directory instead: {grandparent}"
            )
        raise FileNotFoundError(default_message)


def _format_precision_config(
    config: dict[str, Any],
    precision: str,
    loss_scale: float,
    loss_scale_window: int,
    min_loss_scale: int,
    initial_scale_power: int,
    hysteresis: int,
) -> None:
    if "fp16" not in config and precision in ("16-mixed", "16-true"):
        # FP16 is a DeepSpeed standalone AMP implementation
        rank_zero_info("Enabling DeepSpeed FP16. Model parameters and inputs will be cast to `float16`.")
        config["fp16"] = {
            "enabled": True,
            "loss_scale": loss_scale,
            "initial_scale_power": initial_scale_power,

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Inspect the directory: a DeepSpeed checkpoint contains a checkpoint/ subfolder with zero_to_* or mp_rank_* files
2. Re-save or regenerate the checkpoint with DeepSpeed/Fabric if it came from another framework
3. Point at the correct tag directory (e.g. .../lightning_logs/version_0/checkpoint/global_step10)
4. If the file is a single consolidated checkpoint, use DeepSpeedStrategy(..., load_full_weights=True)

Example fix

# before
fabric.load_checkpoint("./my_run", state)  # not a DS checkpoint

# after
fabric.load_checkpoint("./my_run/checkpoint/global_step10", state)

from pathlib import Path
def is_ds_ckpt_dir(p: str) -> bool:
    p = Path(p)
    return (p / "checkpoint").is_dir() or p.name.startswith("global_step")
assert is_ds_ckpt_dir(ckpt_path), f"not a DeepSpeed checkpoint: {ckpt_path}"

try:
    fabric.load_checkpoint(path, state)
except FileNotFoundError as e:
    if "not a valid DeepSpeed checkpoint" in str(e):
        # search save dir for tag folders
        candidates = sorted(Path(save_dir).glob("**/global_step*"))
        ...

When it happens

Trigger: load_checkpoint with an arbitrary directory, a non-checkpoint file, a nonexistent-but-directory-like path, or an empty/failed save directory while using DeepSpeedStrategy.

Common situations: Resuming from a checkpoint saved by a different strategy or plain torch.save; a crashed/interrupted save leaving an incomplete directory; passing the run dir instead of the checkpoint dir; wrong remote bucket path.

Related errors

• {default_message}. It looks like you passed the path to a su
• {default_message}. It looks like you passed the path to a fi
• DeepSpeed was unable to load the checkpoint. Ensure you pass
• You passed in a path to a DeepSpeed config but the path does
• Got FSDPStrategy.load_checkpoint(..., state={state!r}) but a

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/0cc893e556d6011f. Report an issue: GitHub.