{"record":{"id":"ac144099ab0ad5f8","repo":"Lightning-AI/pytorch-lightning","slug":"found-multiple-deepspeed-engine-modules-in-the-giv","errorCode":null,"errorMessage":"Found multiple DeepSpeed engine modules in the given state. Saving checkpoints with DeepSpeed is currently limited to a single model per checkpoint. To save multiple models, call the save method for each model separately with a different path.","messagePattern":"Found multiple DeepSpeed engine modules in the given state\\. Saving checkpoints with DeepSpeed is currently limited to a single model per checkpoint\\. To save multiple models, call the save method for each model separately with a different path\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/lightning/fabric/strategies/deepspeed.py","lineNumber":449,"sourceCode":"            raise TypeError(\n                \"`DeepSpeedStrategy.save_checkpoint(..., storage_options=...)` is not supported because\"\n                \" `DeepSpeedStrategy` does not use the `CheckpointIO`.\"\n            )\n        if filter is not None:\n            raise TypeError(\n                \"`DeepSpeedStrategy.save_checkpoint(..., filter=...)` is not supported because\"\n                \" `DeepSpeedStrategy` manages the state serialization internally.\"\n            )\n\n        engines = _get_deepspeed_engines_from_state(state)\n        if len(engines) == 0:\n            raise ValueError(\n                \"Could not find a DeepSpeed model in the provided checkpoint state. Please provide the model as\"\n                \" part of the state like so: `save_checkpoint(..., state={'model': model, ...})`. Make sure\"\n                \" you set up the model (and optimizers if any) through the strategy before saving the checkpoint.\"\n            )\n        if len(engines) > 1:\n            raise ValueError(\n                \"Found multiple DeepSpeed engine modules in the given state. Saving checkpoints with DeepSpeed is\"\n                \" currently limited to a single model per checkpoint. To save multiple models, call the\"\n                \" save method for each model separately with a different path.\"\n            )\n        engine = engines[0]\n\n        # broadcast the path from rank 0 to ensure all the states are saved in a common path\n        path = self.broadcast(path)\n\n        # split the checkpoint into two parts:\n        # 1) the deepspeed engine encapsulating both the model and optionally the optimizer(s)\n        # 2) the rest of the user's state, which in deepspeed is called `client state`\n        excluded_objects = (engine, engine.optimizer) if engine.optimizer is not None else (engine,)\n        state = {k: v for k, v in state.items() if v not in excluded_objects}\n        _validate_state_keys(state)\n        # there might be other stateful objects unrelated to the deepspeed engine - convert them to a state_dict\n        state = self._convert_stateful_objects_in_state(state, filter={})\n        # use deepspeed's internal checkpointing function to handle partitioned weights across processes","sourceCodeStart":431,"sourceCodeEnd":467,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/fabric/strategies/deepspeed.py#L431-L467","documentation":"A DeepSpeed checkpoint maps to exactly one DeepSpeedEngine. If the state passed to save_checkpoint contains multiple set-up models (multiple engines), saving is ambiguous and this ValueError is raised, suggesting separate save calls per model.","triggerScenarios":"fabric.save_checkpoint(path, {'model_a': m1, 'model_b': m2}) where both m1 and m2 were wrapped by DeepSpeedStrategy engines.","commonSituations":"Multi-model pipelines (e.g. encoder+decoder, GANs) under one DeepSpeed strategy instance; Mixture-of-Experts setups with several engines.","solutions":["Call fabric.save_checkpoint separately for each model with different paths","Or use separate Fabric/strategy instances per model from the start","Consolidate into a single model (e.g. one module containing submodules) if a single checkpoint is required"],"exampleFix":"# before\nfabric.save_checkpoint(path, state={'enc': enc, 'dec': dec})\n# after\nfabric.save_checkpoint(path_enc, state={'enc': enc})\nfabric.save_checkpoint(path_dec, state={'dec': dec})","handlingStrategy":"validation","validationCode":"from deepspeed import DeepSpeedEngine\nengines = [v for v in state.values() if isinstance(v, DeepSpeedEngine)]\nif len(engines) > 1:\n    for name, model in state.items():\n        if isinstance(model, DeepSpeedEngine):\n            fabric.save_checkpoint(f\"{path}_{name}\", {name: model})","typeGuard":"from deepspeed import DeepSpeedEngine\n\ndef single_engine_state(state: dict) -> bool:\n    return sum(isinstance(v, DeepSpeedEngine) for v in state.values()) == 1","tryCatchPattern":null,"preventionTips":["One save_checkpoint call per DeepSpeed model","Design multi-model pipelines with separate strategies/paths"],"tags":["deepspeed","checkpointing","multiple-models","pytorch-lightning"],"backgroundTag":"invalid-state-argument","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}