{"record":{"id":"f67c612a8afca0aa","repo":"huggingface/transformers","slug":"save-pretrained-distributed-checkpoint-true","errorCode":null,"errorMessage":"save_pretrained(..., distributed_checkpoint=True) requires torch>=2.7.","messagePattern":"save_pretrained\\(\\.\\.\\., distributed_checkpoint=True\\) requires torch>=2\\.7\\.","errorType":"exception","errorClass":"OSError","httpStatus":null,"severity":"error","filePath":"src/transformers/distributed/mixin.py","lineNumber":233,"sourceCode":"            save_on_this_rank = save_on_this_rank and _get_torch_distributed_rank() == 0\n        return save_on_this_rank\n\n    def save_distributed_checkpoint(\n        self,\n        model_to_save,\n        save_directory: str | os.PathLike,\n        *,\n        push_to_hub: bool = False,\n        save_on_this_rank: bool = True,\n        repo_id: str | None = None,\n        files_timestamps: dict | None = None,\n        commit_message: str | None = None,\n        token: str | bool | None = None,\n        create_pr: bool = False,\n    ) -> None:\n        \"\"\"Save an FSDP-wrapped model via DCP and optionally push to the Hub.\"\"\"\n        if not is_torch_greater_or_equal(\"2.7\"):\n            raise OSError(\"save_pretrained(..., distributed_checkpoint=True) requires torch>=2.7.\")\n        if not is_fsdp_managed_module(model_to_save):\n            raise ValueError(\n                \"save_pretrained(..., distributed_checkpoint=True) is only supported for FSDP-wrapped models.\"\n            )\n        if getattr(model_to_save, \"_device_mesh\", None) is None:\n            raise ValueError(\n                \"save_pretrained(..., distributed_checkpoint=True) requires the model to have been \"\n                \"initialized with a distributed_config (_device_mesh is None).\"\n            )\n        save_model_checkpoint_distributed(model_to_save, save_directory)\n\n        if push_to_hub and save_on_this_rank:\n            model_card = create_and_tag_model_card(repo_id, self.model_tags, token=token)\n            model_card.save(os.path.join(save_directory, \"README.md\"))\n            self._upload_modified_files(\n                save_directory,\n                repo_id,\n                files_timestamps,","sourceCodeStart":215,"sourceCodeEnd":251,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/distributed/mixin.py#L215-L251","documentation":"The distributed-checkpoint save path (DCP + HuggingFaceStorageWriter) used by save_pretrained(..., distributed_checkpoint=True) relies on APIs that only exist in torch 2.7 and newer. On older torch the save raises OSError before touching the disk, so no partial checkpoint is written.","triggerScenarios":"Calling Model.save_pretrained(..., distributed_checkpoint=True) (or the mixin's save_pretrained_distributed) in an environment where importlib metadata reports torch < 2.7.","commonSituations":"Training environments pinned to torch 2.4-2.6 (common for older CUDA images); CI images lagging behind; upgrading transformers without upgrading torch.","solutions":["Upgrade torch: pip install -U 'torch>=2.7'.","If torch cannot be upgraded, save with the regular gathered path (distributed_checkpoint=False) which full-state-dict gathers to rank 0.","Pin transformers/torch together in requirements to avoid silent version drift."],"exampleFix":"# before\ntorch==2.6.0  # requirements.txt, save_pretrained(..., distributed_checkpoint=True) raises OSError\n\n# after\ntorch>=2.7.0  # requirements.txt","handlingStrategy":"validation","validationCode":"from transformers.utils import is_torch_greater_or_equal\n\ndef assert_torch_for_distributed_save() -> None:\n    if not is_torch_greater_or_equal(\"2.7\"):\n        raise RuntimeError(\"distributed_checkpoint=True needs torch>=2.7; upgrade or save gathered instead\")","typeGuard":null,"tryCatchPattern":"try:\n    model.save_pretrained(out, distributed_checkpoint=True)\nexcept OSError as e:\n    if \"requires torch>=2.7\" in str(e):\n        model.save_pretrained(out)  # fallback: gathered, rank-0 save\n    else:\n        raise","preventionTips":["Assert the torch version at trainer startup, not at save time after hours of training.","Pin 'torch>=2.7' in the environment spec when using distributed checkpoints.","Wrap the save with a fallback to the non-distributed path if you support mixed fleets."],"tags":["torch-version","distributed","checkpointing","environment"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}