Lightning-AI/pytorch-lightning · error · ValueError

Found multiple DeepSpeed engine modules in the given state.

Error message

Found multiple DeepSpeed engine modules in the given state. Saving and loading checkpoints with DeepSpeed is currently limited to a single model per checkpoint. To load multiple model states, call the load method for each model checkpoint separately.

What it means

DeepSpeed checkpoint loading, like saving, supports exactly one engine per checkpoint. If state contains multiple DeepSpeed-set-up models, loading is ambiguous and this ValueError asks you to load each model checkpoint separately.

Source

Thrown at src/lightning/fabric/strategies/deepspeed.py:522

            return super().load_checkpoint(path=path, state=state, strict=strict, weights_only=weights_only)

        if not state:
            raise ValueError(
                f"Got DeepSpeedStrategy.load_checkpoint(..., state={state!r}) but a state with at least "
                f" a model instance to reload is required. Pass it in like so:"
                " DeepSpeedStrategy.load_checkpoint(..., state={'model': model, ...})"
            )
        _validate_checkpoint_directory(path)

        engines = _get_deepspeed_engines_from_state(state)
        if len(engines) == 0:
            raise ValueError(
                "Could not find a DeepSpeed model in the provided checkpoint state. Please provide the model as"
                " part of the state like so: `load_checkpoint(..., state={'model': model, ...})`. Make sure"
                " you set up the model (and optimizers if any) through the strategy before loading the checkpoint."
            )
        if len(engines) > 1:
            raise ValueError(
                "Found multiple DeepSpeed engine modules in the given state. Saving and loading checkpoints"
                " with DeepSpeed is currently limited to a single model per checkpoint. To load multiple model"
                " states, call the load method for each model checkpoint separately."
            )
        engine = engines[0]

        from deepspeed.runtime.base_optimizer import DeepSpeedOptimizer

        optimzer_state_requested = any(isinstance(item, (Optimizer, DeepSpeedOptimizer)) for item in state.values())

        torch.cuda.empty_cache()
        _, client_state = engine.load_checkpoint(
            path,
            tag="checkpoint",
            load_optimizer_states=optimzer_state_requested,
            load_lr_scheduler_states=False,
            load_module_strict=strict,
        )

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Load each model's checkpoint in a separate call with its own path and single-model state
2. Split models across separate Fabric/DeepSpeed strategy instances so each has one engine
3. Restructure into a single module if a single-engine checkpoint is acceptable

Example fix

# before
fabric.load_checkpoint(path, state={'m1': m1, 'm2': m2})
# after
fabric.load_checkpoint(path1, state={'m1': m1})
fabric.load_checkpoint(path2, state={'m2': m2})

from deepspeed import DeepSpeedEngine
engines = [v for v in state.values() if isinstance(v, DeepSpeedEngine)]
if len(engines) > 1:
    for name, model in state.items():
        if isinstance(model, DeepSpeedEngine):
            fabric.load_checkpoint(f"{path}_{name}", {name: model})

from deepspeed import DeepSpeedEngine

def single_engine_state(state: dict) -> bool:
    return sum(isinstance(v, DeepSpeedEngine) for v in state.values()) == 1

When it happens

Trigger: fabric.load_checkpoint(path, state={'m1': model1, 'm2': model2}) where both models were wrapped as DeepSpeed engines by the strategy.

Common situations: Multi-model training (autoencoders, GANs) trying to resume everything in one call; per-model checkpoints being merged into a single load call.

Related errors

• Found multiple DeepSpeed engine modules in the given state.
• Got DeepSpeedStrategy.load_checkpoint(..., state={state!r})
• Could not find a DeepSpeed model in the provided checkpoint
• `DeepSpeedStrategy.save_checkpoint(..., storage_options=...)
• `DeepSpeedStrategy.save_checkpoint(..., filter=...)` is not

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/800898f0a70cb0de. Report an issue: GitHub.