Lightning-AI/pytorch-lightning · error · ValueError

Got DeepSpeedStrategy.load_checkpoint(..., state={state!r})

Error message

Got DeepSpeedStrategy.load_checkpoint(..., state={state!r}) but a state with at least  a model instance to reload is required. Pass it in like so: DeepSpeedStrategy.load_checkpoint(..., state={'model': model, ...})

What it means

Loading a DeepSpeed checkpoint requires state to restore into — at minimum a set-up model. DeepSpeedStrategy.load_checkpoint rejects an empty/None state ({} or None) because there is nothing to restore the checkpoint into; unlike some strategies it cannot load into a blank slate.

Source

Thrown at src/lightning/fabric/strategies/deepspeed.py:507

            Dictionary with the state inside DeepSpeed's engine

        Raises:
            ValueError:
                If no state is provided, when no :class:`deepspeed.DeepSpeedEngine` objects were found in the
                state, or when multiple :class:`deepspeed.DeepSpeedEngine` objects were found.
            RuntimeError:
                If DeepSpeed was unable to load the checkpoint due to missing files or because the checkpoint is
                not in the expected DeepSpeed format.

        """
        if isinstance(state, (Module, Optimizer)) or self.load_full_weights and self.zero_stage_3:
            # This code path to enables loading a checkpoint from a non-deepspeed checkpoint or from
            # a consolidated checkpoint
            path = self.broadcast(path)
            return super().load_checkpoint(path=path, state=state, strict=strict, weights_only=weights_only)

        if not state:
            raise ValueError(
                f"Got DeepSpeedStrategy.load_checkpoint(..., state={state!r}) but a state with at least "
                f" a model instance to reload is required. Pass it in like so:"
                " DeepSpeedStrategy.load_checkpoint(..., state={'model': model, ...})"
            )
        _validate_checkpoint_directory(path)

        engines = _get_deepspeed_engines_from_state(state)
        if len(engines) == 0:
            raise ValueError(
                "Could not find a DeepSpeed model in the provided checkpoint state. Please provide the model as"
                " part of the state like so: `load_checkpoint(..., state={'model': model, ...})`. Make sure"
                " you set up the model (and optimizers if any) through the strategy before loading the checkpoint."
            )
        if len(engines) > 1:
            raise ValueError(
                "Found multiple DeepSpeed engine modules in the given state. Saving and loading checkpoints"
                " with DeepSpeed is currently limited to a single model per checkpoint. To load multiple model"
                " states, call the load method for each model checkpoint separately."

View on GitHub (pinned to 9fed5c27d2)

Solutions

  1. Set up the model (and optimizer) first, then load: fabric.load_checkpoint(path, state={'model': model, 'optimizer': optimizer})
  2. Remember load_checkpoint restores into your objects in place and returns the state back
  3. Populate state from your training objects before calling load

Example fix

# before
state = fabric.load_checkpoint(path, state={})
# after
model, optimizer = fabric.setup(model, optimizer)
state = {'model': model, 'optimizer': optimizer}
fabric.load_checkpoint(path, state)
Defensive patterns

Strategy: validation

Validate before calling

if not state or not any(v is not None for v in state.values()):
    raise ValueError("populate state with the set-up model before loading")
fabric.load_checkpoint(path, state)

Type guard

def load_state_valid(state) -> bool:
    return bool(state) and any(v is not None for v in state.values())

Prevention

When it happens

Trigger: fabric.load_checkpoint(path, state={}) or fabric.load_checkpoint(path) under DeepSpeedStrategy (omitted/empty state).

Common situations: Generic resume helpers that call load_checkpoint(path, state) before populating state; assuming the strategy returns a loaded state rather than filling one in-place.

Related errors

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/efee26e14c50dc3a. Report an issue: GitHub.