{"record":{"id":"6e17d15c4e7f5b9e","repo":"Lightning-AI/pytorch-lightning","slug":"found-multiple-fsdp-models-in-the-given-state-sav","errorCode":null,"errorMessage":"Found multiple FSDP models in the given state. Saving checkpoints with FSDP is currently limited to a single model per checkpoint. To save multiple models, call the save method for each model separately with a different path.","messagePattern":"Found multiple FSDP models in the given state\\. Saving checkpoints with FSDP is currently limited to a single model per checkpoint\\. To save multiple models, call the save method for each model separately with a different path\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/lightning/fabric/strategies/fsdp.py","lineNumber":466,"sourceCode":"                \"FSDP doesn't support loading sharded filtered checkpoints, so saving them is disabled.\"\n            )\n\n        # broadcast the path from rank 0 to ensure all the states are saved in a common path\n        path = _resolve_path(self.broadcast(path))\n        if self._state_dict_type == \"full\" and _is_checkpoint_dir(path) and not _is_sharded_checkpoint(path):\n            raise IsADirectoryError(f\"The checkpoint path exists and is a directory: {path}\")\n\n        from torch.distributed.fsdp import FullyShardedDataParallel as FSDP\n\n        modules = [module for module in state.values() if _has_fsdp_modules(module)]\n        if len(modules) == 0:\n            raise ValueError(\n                \"Could not find a FSDP model in the provided checkpoint state. Please provide the model as\"\n                \" part of the state like so: `save_checkpoint(..., state={'model': model, ...})`. Make sure\"\n                \" you set up the model (and optimizers if any) through the strategy before saving the checkpoint.\"\n            )\n        if len(modules) > 1:\n            raise ValueError(\n                \"Found multiple FSDP models in the given state. Saving checkpoints with FSDP is\"\n                \" currently limited to a single model per checkpoint. To save multiple models, call the\"\n                \" save method for each model separately with a different path.\"\n            )\n        module = modules[0]\n\n        if self._state_dict_type == \"sharded\":\n            _prepare_directory_checkpoint(path)\n\n            state_dict_ctx = _get_sharded_state_dict_context(module)\n\n            # replace the modules and optimizer objects in the state with their local state dict\n            # and separate the user's metadata\n            converted_state: dict[str, Any] = {}\n            metadata: dict[str, Any] = {}\n            with state_dict_ctx:\n                for key, obj in state.items():\n                    converted: Any","sourceCodeStart":448,"sourceCodeEnd":484,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/fabric/strategies/fsdp.py#L448-L484","documentation":"The FSDP strategy's save_checkpoint implementation only supports checkpointing one FSDP-wrapped model at a time. When it scans the state dict for FSDP modules it finds more than one, it refuses to save because sharded checkpoint metadata cannot unambiguously represent multiple independent FSDP root modules.","triggerScenarios":"Calling fabric.save(...) / strategy.save_checkpoint(path, state) where state is a dict containing two or more modules that contain FullyShardedDataParallel-wrapped submodules, e.g. {'model': fsdp_model, 'vae': fsdp_vae}.","commonSituations":"Diffusion-style training with a UNet + text encoder + VAE all wrapped in FSDP in one Fabric setup; stacking a student and teacher model in a single state dict; migrating from DDP where multi-model checkpoints worked fine.","solutions":["Split the save into separate calls: save_checkpoint(path1, {'model': model1}) and save_checkpoint(path2, {'model': model2})","Keep only one FSDP model in the state and pass the other unwrapped or handle it manually","Upgrade to torch.distributed.checkpoint usage directly if you truly need one file for multiple sharded models"],"exampleFix":"# before\nfabric.save('ckpt', {'unet': unet, 'vae': vae})\n# after\nfabric.save('ckpt/unet', {'unet': unet})\nfabric.save('ckpt/vae', {'vae': vae})","handlingStrategy":"validation","validationCode":"from torch.distributed.fsdp import FullyShardedDataParallel as FSDP\n\ndef count_fsdp_models(state: dict) -> int:\n    def has_fsdp(m):\n        return isinstance(m, FSDP) or any(isinstance(mod, FSDP) for mod in m.modules()) if hasattr(m, 'modules') else False\n    return sum(1 for v in state.values() if has_fsdp(v))\n\nif count_fsdp_models(state) > 1:\n    for key, val in state.items():\n        fabric.save(f'{out_dir}/{key}', {key: val})","typeGuard":"def is_fsdp_model(obj) -> bool:\n    from torch.distributed.fsdp import FullyShardedDataParallel as FSDP\n    return isinstance(obj, FSDP) or (hasattr(obj, 'modules') and any(isinstance(m, FSDP) for m in obj.modules()))","tryCatchPattern":null,"preventionTips":["Keep one FSDP model per checkpoint path","Structure multi-model setups with a helper that saves each model under its own subdirectory"],"tags":["fsdp","checkpoint","save","pytorch-lightning","distributed"],"backgroundTag":"fsdp-multi-model-checkpoint","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}