{"record":{"id":"d0e2660331df6079","repo":"huggingface/transformers","slug":"save-pretrained-distributed-checkpoint-true-d0e266","errorCode":null,"errorMessage":"save_pretrained(..., distributed_checkpoint=True) is only supported for FSDP-wrapped models.","messagePattern":"save_pretrained\\(\\.\\.\\., distributed_checkpoint=True\\) is only supported for FSDP-wrapped models\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/distributed/mixin.py","lineNumber":235,"sourceCode":"\n    def save_distributed_checkpoint(\n        self,\n        model_to_save,\n        save_directory: str | os.PathLike,\n        *,\n        push_to_hub: bool = False,\n        save_on_this_rank: bool = True,\n        repo_id: str | None = None,\n        files_timestamps: dict | None = None,\n        commit_message: str | None = None,\n        token: str | bool | None = None,\n        create_pr: bool = False,\n    ) -> None:\n        \"\"\"Save an FSDP-wrapped model via DCP and optionally push to the Hub.\"\"\"\n        if not is_torch_greater_or_equal(\"2.7\"):\n            raise OSError(\"save_pretrained(..., distributed_checkpoint=True) requires torch>=2.7.\")\n        if not is_fsdp_managed_module(model_to_save):\n            raise ValueError(\n                \"save_pretrained(..., distributed_checkpoint=True) is only supported for FSDP-wrapped models.\"\n            )\n        if getattr(model_to_save, \"_device_mesh\", None) is None:\n            raise ValueError(\n                \"save_pretrained(..., distributed_checkpoint=True) requires the model to have been \"\n                \"initialized with a distributed_config (_device_mesh is None).\"\n            )\n        save_model_checkpoint_distributed(model_to_save, save_directory)\n\n        if push_to_hub and save_on_this_rank:\n            model_card = create_and_tag_model_card(repo_id, self.model_tags, token=token)\n            model_card.save(os.path.join(save_directory, \"README.md\"))\n            self._upload_modified_files(\n                save_directory,\n                repo_id,\n                files_timestamps,\n                commit_message=commit_message,\n                token=token,","sourceCodeStart":217,"sourceCodeEnd":253,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/distributed/mixin.py#L217-L253","documentation":"The distributed checkpoint save path only knows how to serialize FSDP2-managed (fully_shard-wrapped) modules via is_fsdp_managed_module. If the model was never wrapped — because fsdp_size was 1 or the distributed config was not applied — the writer cannot map sharded parameters and raises ValueError.","triggerScenarios":"Calling save_pretrained(..., distributed_checkpoint=True) on a plain model: loaded with from_pretrained without distributed_config, or with a config where fsdp_size=1 (TP-only or no sharding).","commonSituations":"Trying the new save flag on a TP-only model; saving after manually unwrapping FSDP; calling the FSDP save helper on a checkpoint-gathered model.","solutions":["Initialize the model with distributed_config={'fsdp_size': N>1, ...} so from_pretrained wraps it with fully_shard, then save.","If the model is intentionally not FSDP-sharded, use the standard save_pretrained (distributed_checkpoint omitted/False).","Verify wrapping before saving: assert any(is_fsdp_managed_module(m) for m in model.modules())."],"exampleFix":"# before\nmodel = AutoModelForCausalLM.from_pretrained(model_id)  # no fsdp\nmodel.save_pretrained(out_dir, distributed_checkpoint=True)  # raises\n\n# after\nmodel = AutoModelForCausalLM.from_pretrained(model_id, distributed_config={\"fsdp_size\": world_size})\nmodel.save_pretrained(out_dir, distributed_checkpoint=True)","handlingStrategy":"validation","validationCode":"from transformers.distributed.utils import is_fsdp_managed_module\n\ndef can_distributed_save(model) -> bool:\n    return any(is_fsdp_managed_module(m) for m in model.modules())","typeGuard":"def is_fsdp_model(model) -> bool:\n    return any(is_fsdp_managed_module(m) for m in model.modules())","tryCatchPattern":"try:\n    model.save_pretrained(out, distributed_checkpoint=True)\nexcept ValueError as e:\n    if \"only supported for FSDP-wrapped\" in str(e):\n        model.save_pretrained(out)  # plain save for non-FSDP models\n    else:\n        raise","preventionTips":["Always load with distributed_config={'fsdp_size': N} when you plan distributed saves.","Branch save logic on whether the model is FSDP-managed.","Keep one code path per sharding strategy; do not mix flags."],"tags":["fsdp","distributed","checkpointing","config"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}