{"record":{"id":"800898f0a70cb0de","repo":"Lightning-AI/pytorch-lightning","slug":"found-multiple-deepspeed-engine-modules-in-the-giv-800898","errorCode":null,"errorMessage":"Found multiple DeepSpeed engine modules in the given state. Saving and loading checkpoints with DeepSpeed is currently limited to a single model per checkpoint. To load multiple model states, call the load method for each model checkpoint separately.","messagePattern":"Found multiple DeepSpeed engine modules in the given state\\. Saving and loading checkpoints with DeepSpeed is currently limited to a single model per checkpoint\\. To load multiple model states, call the load method for each model checkpoint separately\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/lightning/fabric/strategies/deepspeed.py","lineNumber":522,"sourceCode":"            return super().load_checkpoint(path=path, state=state, strict=strict, weights_only=weights_only)\n\n        if not state:\n            raise ValueError(\n                f\"Got DeepSpeedStrategy.load_checkpoint(..., state={state!r}) but a state with at least \"\n                f\" a model instance to reload is required. Pass it in like so:\"\n                \" DeepSpeedStrategy.load_checkpoint(..., state={'model': model, ...})\"\n            )\n        _validate_checkpoint_directory(path)\n\n        engines = _get_deepspeed_engines_from_state(state)\n        if len(engines) == 0:\n            raise ValueError(\n                \"Could not find a DeepSpeed model in the provided checkpoint state. Please provide the model as\"\n                \" part of the state like so: `load_checkpoint(..., state={'model': model, ...})`. Make sure\"\n                \" you set up the model (and optimizers if any) through the strategy before loading the checkpoint.\"\n            )\n        if len(engines) > 1:\n            raise ValueError(\n                \"Found multiple DeepSpeed engine modules in the given state. Saving and loading checkpoints\"\n                \" with DeepSpeed is currently limited to a single model per checkpoint. To load multiple model\"\n                \" states, call the load method for each model checkpoint separately.\"\n            )\n        engine = engines[0]\n\n        from deepspeed.runtime.base_optimizer import DeepSpeedOptimizer\n\n        optimzer_state_requested = any(isinstance(item, (Optimizer, DeepSpeedOptimizer)) for item in state.values())\n\n        torch.cuda.empty_cache()\n        _, client_state = engine.load_checkpoint(\n            path,\n            tag=\"checkpoint\",\n            load_optimizer_states=optimzer_state_requested,\n            load_lr_scheduler_states=False,\n            load_module_strict=strict,\n        )","sourceCodeStart":504,"sourceCodeEnd":540,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/fabric/strategies/deepspeed.py#L504-L540","documentation":"DeepSpeed checkpoint loading, like saving, supports exactly one engine per checkpoint. If state contains multiple DeepSpeed-set-up models, loading is ambiguous and this ValueError asks you to load each model checkpoint separately.","triggerScenarios":"fabric.load_checkpoint(path, state={'m1': model1, 'm2': model2}) where both models were wrapped as DeepSpeed engines by the strategy.","commonSituations":"Multi-model training (autoencoders, GANs) trying to resume everything in one call; per-model checkpoints being merged into a single load call.","solutions":["Load each model's checkpoint in a separate call with its own path and single-model state","Split models across separate Fabric/DeepSpeed strategy instances so each has one engine","Restructure into a single module if a single-engine checkpoint is acceptable"],"exampleFix":"# before\nfabric.load_checkpoint(path, state={'m1': m1, 'm2': m2})\n# after\nfabric.load_checkpoint(path1, state={'m1': m1})\nfabric.load_checkpoint(path2, state={'m2': m2})","handlingStrategy":"validation","validationCode":"from deepspeed import DeepSpeedEngine\nengines = [v for v in state.values() if isinstance(v, DeepSpeedEngine)]\nif len(engines) > 1:\n    for name, model in state.items():\n        if isinstance(model, DeepSpeedEngine):\n            fabric.load_checkpoint(f\"{path}_{name}\", {name: model})","typeGuard":"from deepspeed import DeepSpeedEngine\n\ndef single_engine_state(state: dict) -> bool:\n    return sum(isinstance(v, DeepSpeedEngine) for v in state.values()) == 1","tryCatchPattern":null,"preventionTips":["Resume each DeepSpeed model with its own checkpoint path","Keep per-model checkpoint path registries in multi-model trainers"],"tags":["deepspeed","checkpointing","resume","multiple-models","pytorch-lightning"],"backgroundTag":"invalid-state-argument","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}