Lightning-AI/pytorch-lightning · critical · MisconfigurationException

DeepSpeed was unable to load the checkpoint. Ensure you pass

Error message

DeepSpeed was unable to load the checkpoint. Ensure you passed in a DeepSpeed compatible checkpoint or a single checkpoint file with `Trainer(strategy=DeepSpeedStrategy(load_full_weights=True))`.

What it means

After DeepSpeed's `load_checkpoint` returns, a None client_state means DeepSpeed did not recognize the path as a checkpoint it created (its engine returns the user-state dict only for its own tag-based checkpoints). Lightning surfaces this as MisconfigurationException telling you to either pass a DeepSpeed checkpoint or enable load_full_weights to read a consolidated file.

Source

Thrown at src/lightning/pytorch/strategies/deepspeed.py:700

            return super().load_checkpoint(checkpoint_path, weights_only)

        _validate_checkpoint_directory(checkpoint_path)

        # Rely on deepspeed to load the checkpoint and necessary information
        assert self.lightning_module is not None

        from lightning.pytorch.trainer.states import TrainerFn

        is_fitting = self.lightning_module.trainer.state.fn == TrainerFn.FITTING

        _, client_state = self.deepspeed_engine.load_checkpoint(
            checkpoint_path,
            load_optimizer_states=is_fitting,
            load_lr_scheduler_states=False,
            load_module_strict=self.lightning_module.strict_loading,
        )
        if client_state is None:
            raise MisconfigurationException(
                "DeepSpeed was unable to load the checkpoint. Ensure you passed in a DeepSpeed compatible checkpoint "
                "or a single checkpoint file with `Trainer(strategy=DeepSpeedStrategy(load_full_weights=True))`."
            )
        return client_state

    @property
    @override
    def lightning_restore_optimizer(self) -> bool:
        assert self.lightning_module is not None
        # managed by DeepSpeed
        if self.load_full_weights and self.zero_stage_3 and self.lightning_module.trainer.state.fn == TrainerFn.FITTING:
            rank_zero_warn(
                "A single checkpoint file has been given. This means optimizer states cannot be restored."
                " If you'd like to restore these states, you must provide a path to the originally saved DeepSpeed"
                " checkpoint. When using ZeRO 3, the original path should be a directory."
            )
        return False

View on GitHub (pinned to 9fed5c27d2)

Solutions

If resuming from a full single-file Lightning checkpoint, construct `DeepSpeedStrategy(load_full_weights=True)`
If resuming a ZeRO sharded run, point `ckpt_path` at the DeepSpeed checkpoint directory (containing the tags), not just the .ckpt file
Verify the checkpoint was produced by DeepSpeedStrategy (look for `zero/` subfolder, `latest` tag file) before resuming

Example fix

# before
strategy = DeepSpeedStrategy(config=cfg)
trainer = Trainer(strategy=strategy)
trainer.fit(model, ckpt_path="last.ckpt")  # single-file ckpt

# after
strategy = DeepSpeedStrategy(config=cfg, load_full_weights=True)
trainer = Trainer(strategy=strategy)
trainer.fit(model, ckpt_path="last.ckpt")

Defensive patterns

Strategy: try-catch

Validate before calling

from pathlib import Path
def is_deepspeed_ckpt(p):
    p = Path(p)
    return p.is_dir() and (p / "latest").exists()
# choose load mode before resume
full = not is_deepspeed_ckpt(ckpt_path)

Type guard

def is_deepspeed_checkpoint(path: str) -> bool:
    p = Path(path)
    return p.is_dir() and (p / "latest").exists()

Try / catch

from lightning.pytorch.utilities.exceptions import MisconfigurationException
try:
    trainer.fit(model, ckpt_path=ckpt)
except MisconfigurationException as e:
    if "DeepSpeed was unable to load" in str(e):
        trainer.strategy = DeepSpeedStrategy(config=cfg, load_full_weights=True)
        raise SystemExit("Retry with load_full_weights=True")
    raise

Prevention

Always pass the full DeepSpeed checkpoint directory (with tags) for sharded resumes
Use load_full_weights=True when the checkpoint came from another strategy
Keep checkpoint dirs intact — copy whole directories, not single .ckpt files

When it happens

Trigger: Calling `trainer.strategy.load_checkpoint(path)` / `trainer.fit(..., ckpt_path=...)` with DeepSpeedStrategy pointing at (a) a plain Lightning `.ckpt` file while `load_full_weights=False`, (b) a directory missing DeepSpeed's tags (zero-to_fp32.py, latest etc.), or (c) a corrupt/foreign checkpoint dir.

Common situations: Resuming from a checkpoint saved by a different strategy (DDP/FSDP) without `DeepSpeedStrategy(load_full_weights=True)`; copying only the .ckpt file out of a sharded checkpoint dir; path typos resolving to an existing but wrong dir.

Related errors

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/261f92a22bd2c239. Report an issue: GitHub.