Lightning-AI/pytorch-lightning · error · ValueError

Could not find a DeepSpeed model in the provided checkpoint

Error message

Could not find a DeepSpeed model in the provided checkpoint state. Please provide the model as part of the state like so: `save_checkpoint(..., state={'model': model, ...})`. Make sure you set up the model (and optimizers if any) through the strategy before saving the checkpoint.

What it means

DeepSpeed checkpoints are saved by the DeepSpeedEngine, so the state passed to save_checkpoint must contain at least one module that was set up through the strategy (i.e. a DeepSpeedEngine instance). If _get_deepspeed_engines_from_state finds none, this ValueError is raised.

Source

Thrown at src/lightning/fabric/strategies/deepspeed.py:443

            ValueError:
                When no :class:`deepspeed.DeepSpeedEngine` objects were found in the state, or when multiple
                :class:`deepspeed.DeepSpeedEngine` objects were found.

        """
        if storage_options is not None:
            raise TypeError(
                "`DeepSpeedStrategy.save_checkpoint(..., storage_options=...)` is not supported because"
                " `DeepSpeedStrategy` does not use the `CheckpointIO`."
            )
        if filter is not None:
            raise TypeError(
                "`DeepSpeedStrategy.save_checkpoint(..., filter=...)` is not supported because"
                " `DeepSpeedStrategy` manages the state serialization internally."
            )

        engines = _get_deepspeed_engines_from_state(state)
        if len(engines) == 0:
            raise ValueError(
                "Could not find a DeepSpeed model in the provided checkpoint state. Please provide the model as"
                " part of the state like so: `save_checkpoint(..., state={'model': model, ...})`. Make sure"
                " you set up the model (and optimizers if any) through the strategy before saving the checkpoint."
            )
        if len(engines) > 1:
            raise ValueError(
                "Found multiple DeepSpeed engine modules in the given state. Saving checkpoints with DeepSpeed is"
                " currently limited to a single model per checkpoint. To save multiple models, call the"
                " save method for each model separately with a different path."
            )
        engine = engines[0]

        # broadcast the path from rank 0 to ensure all the states are saved in a common path
        path = self.broadcast(path)

        # split the checkpoint into two parts:
        # 1) the deepspeed engine encapsulating both the model and optionally the optimizer(s)
        # 2) the rest of the user's state, which in deepspeed is called `client state`

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Pass the set-up model object in state: fabric.save_checkpoint(path, {'model': model, ...}) where model came from fabric.setup/fabric.setup_module
2. Ensure the same Fabric/strategy instance was used to set up the model
3. For raw-weight-only saves, use your own torch.save instead of the strategy checkpoint API

Example fix

# before
fabric.save_checkpoint(path, state={'weights': model.state_dict()})
# after
model = fabric.setup(model, optimizer)
fabric.save_checkpoint(path, state={'model': model, 'optimizer': optimizer})

from deepspeed import DeepSpeedEngine
def has_engine(state) -> bool:
    return any(isinstance(v, DeepSpeedEngine) for v in state.values())
assert has_engine(state), "state must include the set-up model"

from deepspeed import DeepSpeedEngine

def checkpoint_state_valid(state: dict) -> bool:
    return bool(state) and any(isinstance(v, DeepSpeedEngine) for v in state.values())

When it happens

Trigger: fabric.save_checkpoint(path, state) where state contains only raw tensors, an unwrapped nn.Module, or a module created but never passed through fabric.setup/fabric.setup_module under DeepSpeedStrategy.

Common situations: Saving a state dict of raw weights instead of the model object; calling save_checkpoint before setup; mixing strategy objects (model set up under a different fabric instance).

Related errors

• Got DeepSpeedStrategy.load_checkpoint(..., state={state!r})
• Could not find a DeepSpeed model in the provided checkpoint
• `DeepSpeedStrategy.save_checkpoint(..., storage_options=...)
• `DeepSpeedStrategy.save_checkpoint(..., filter=...)` is not
• Found multiple DeepSpeed engine modules in the given state.

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/d39c7ccb88a00378. Report an issue: GitHub.