Lightning-AI/pytorch-lightning · critical · RuntimeError

DeepSpeed was unable to load the checkpoint. Ensure you pass

Error message

DeepSpeed was unable to load the checkpoint. Ensure you passed in a DeepSpeed compatible checkpoint or a single checkpoint file by setting `DeepSpeedStrategy(..., load_full_weights=True)`.

What it means

DeepSpeedStrategy.load_checkpoint calls DeepSpeed's Engine.load_checkpoint, which returns a client_state dict on success. If DeepSpeed returns None, the checkpoint at the given path was not a DeepSpeed-compatible checkpoint (no loadable checkpoint state was found), so Lightning raises RuntimeError. The usual cause is pointing at a single consolidated .ckpt/.pth file while the strategy is in sharded mode, or a corrupt/foreign directory.

Source

Thrown at src/lightning/fabric/strategies/deepspeed.py:543

                " states, call the load method for each model checkpoint separately."
            )
        engine = engines[0]

        from deepspeed.runtime.base_optimizer import DeepSpeedOptimizer

        optimzer_state_requested = any(isinstance(item, (Optimizer, DeepSpeedOptimizer)) for item in state.values())

        torch.cuda.empty_cache()
        _, client_state = engine.load_checkpoint(
            path,
            tag="checkpoint",
            load_optimizer_states=optimzer_state_requested,
            load_lr_scheduler_states=False,
            load_module_strict=strict,
        )

        if client_state is None:
            raise RuntimeError(
                "DeepSpeed was unable to load the checkpoint. Ensure you passed in a DeepSpeed compatible checkpoint"
                " or a single checkpoint file by setting `DeepSpeedStrategy(..., load_full_weights=True)`."
            )

        # `Engine.load_checkpoint` adds useless keys 'optimizer' and 'lr_scheduler' to the client state; remove
        # them to avoid name collision with user state
        keys = set(client_state) & set(state) - {"optimizer", "lr_scheduler"}
        _move_state_into(source=client_state, destination=state, keys=keys)
        return client_state

    @override
    def clip_gradients_norm(
        self,
        module: "DeepSpeedEngine",
        optimizer: Optimizer,
        max_norm: Union[float, int],
        norm_type: Union[float, int] = 2.0,
        error_if_nonfinite: bool = True,

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Verify the path is a DeepSpeed checkpoint: it should contain a checkpoint/ subfolder with zero_to_* files or a single-file full checkpoint
2. If you have a single consolidated checkpoint file, construct DeepSpeedStrategy(..., load_full_weights=True) so the whole file is loaded directly
3. Re-save the checkpoint from a DeepSpeed run before trying to resume
4. Check that the checkpoint was produced by the same code path (fabric.save_checkpoint) you are now loading with

Example fix

# before
strategy = DeepSpeedStrategy(config=ds_config)
fabric.load_checkpoint("model.ckpt", state)  # RuntimeError

# after
strategy = DeepSpeedStrategy(config=ds_config, load_full_weights=True)
fabric.load_checkpoint("model.ckpt", state)

from lightning.fabric.strategies.deepspeed import _is_deepspeed_checkpoint
from fsspec import filesystem
path = "ckpt/global_step10"
if not _is_deepspeed_checkpoint(path, filesystem("file")):
    if Path(path).is_file():
        strategy = DeepSpeedStrategy(..., load_full_weights=True)  # single-file path
    else:
        raise ValueError(f"{path} is not a DeepSpeed checkpoint")

try:
    fabric.load_checkpoint(path, state)
except RuntimeError as e:
    if "unable to load the checkpoint" in str(e):
        # fall back to full-weights load or re-save from a DeepSpeed run
        ...

When it happens

Trigger: Calling fabric.load_checkpoint(...) or strategy.load_checkpoint(...) with a path that is not a DeepSpeed engine checkpoint (e.g. a plain PyTorch state_dict file, an FSDP checkpoint, or a non-checkpoint path) while using DeepSpeedStrategy without load_full_weights=True.

Common situations: User saved a checkpoint with another strategy or plain torch.save and tries to resume with DeepSpeed; passing the wrong directory (missing the global step subfolder); mixed Lightning version upgrade changing checkpoint layout; remote URI not accessible so DeepSpeed finds nothing.

Related errors

• {default_message}. It looks like you passed the path to a su
• {default_message}. It looks like you passed the path to a fi
• The provided path is not a valid DeepSpeed checkpoint: {path
• Got FSDPStrategy.load_checkpoint(..., state={state!r}) but a
• Could not find a FSDP model in the provided checkpoint state

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/32d5e7227f5cc7dd. Report an issue: GitHub.