{"record":{"id":"d3a43da819b1656e","repo":"huggingface/transformers","slug":"save-pretrained-distributed-checkpoint-true-d3a43d","errorCode":null,"errorMessage":"save_pretrained(..., distributed_checkpoint=True) requires the model to have been initialized with a distributed_config (_device_mesh is None).","messagePattern":"save_pretrained\\(\\.\\.\\., distributed_checkpoint=True\\) requires the model to have been initialized with a distributed_config \\(_device_mesh is None\\)\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/distributed/mixin.py","lineNumber":239,"sourceCode":"        save_directory: str | os.PathLike,\n        *,\n        push_to_hub: bool = False,\n        save_on_this_rank: bool = True,\n        repo_id: str | None = None,\n        files_timestamps: dict | None = None,\n        commit_message: str | None = None,\n        token: str | bool | None = None,\n        create_pr: bool = False,\n    ) -> None:\n        \"\"\"Save an FSDP-wrapped model via DCP and optionally push to the Hub.\"\"\"\n        if not is_torch_greater_or_equal(\"2.7\"):\n            raise OSError(\"save_pretrained(..., distributed_checkpoint=True) requires torch>=2.7.\")\n        if not is_fsdp_managed_module(model_to_save):\n            raise ValueError(\n                \"save_pretrained(..., distributed_checkpoint=True) is only supported for FSDP-wrapped models.\"\n            )\n        if getattr(model_to_save, \"_device_mesh\", None) is None:\n            raise ValueError(\n                \"save_pretrained(..., distributed_checkpoint=True) requires the model to have been \"\n                \"initialized with a distributed_config (_device_mesh is None).\"\n            )\n        save_model_checkpoint_distributed(model_to_save, save_directory)\n\n        if push_to_hub and save_on_this_rank:\n            model_card = create_and_tag_model_card(repo_id, self.model_tags, token=token)\n            model_card.save(os.path.join(save_directory, \"README.md\"))\n            self._upload_modified_files(\n                save_directory,\n                repo_id,\n                files_timestamps,\n                commit_message=commit_message,\n                token=token,\n                create_pr=create_pr,\n            )\n\n    def gather_sharded_state_dict_for_save(","sourceCodeStart":221,"sourceCodeEnd":257,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/distributed/mixin.py#L221-L257","documentation":"The distributed save path needs the device mesh that transformers builds when the model is prepared with a distributed_config. If model._device_mesh is None the model bypassed distributed initialization (freshly constructed, loaded without distributed_config, or a re-wrapped copy), and the save cannot proceed — the DCP writer has no mesh to coordinate ranks over.","triggerScenarios":"Constructing the model via __init__ instead of from_pretrained(distributed_config=...); loading with distributed_config=None; deep-copying or re-instantiating the model after training and calling save_pretrained(..., distributed_checkpoint=True) on the copy.","commonSituations":"Fine-tuning scripts that build models manually; wrapping the raw model with torch's own fully_shard afterwards (which does not set _device_mesh); saving an EMA/eval copy of the model.","solutions":["Load the model through from_pretrained(..., distributed_config={'fsdp_size': N, 'tp_size': M}) so _device_mesh is set.","If you wrapped FSDP yourself, unwrap and re-load via the transformers distributed path before distributed save.","For ad-hoc copies, fall back to the non-distributed save on rank 0 after gathering the full state dict."],"exampleFix":"# before\nmodel = AutoModelForCausalLM.from_pretrained(model_id)  # _device_mesh is None\nmodel.save_pretrained(out_dir, distributed_checkpoint=True)  # raises\n\n# after\nmodel = AutoModelForCausalLM.from_pretrained(model_id, distributed_config={\"fsdp_size\": 4})\nmodel.save_pretrained(out_dir, distributed_checkpoint=True)","handlingStrategy":"validation","validationCode":"def has_device_mesh(model) -> bool:\n    return getattr(model, \"_device_mesh\", None) is not None","typeGuard":"def is_distributed_initialized_model(model) -> bool:\n    return getattr(model, \"_device_mesh\", None) is not None and getattr(model, \"_tp_plan\", None) is not None","tryCatchPattern":"try:\n    model.save_pretrained(out, distributed_checkpoint=True)\nexcept ValueError as e:\n    if \"_device_mesh is None\" in str(e):\n        model = Model.from_pretrained(model_id, distributed_config=cfg)  # reload via distributed path\n        model.save_pretrained(out, distributed_checkpoint=True)\n    else:\n        raise","preventionTips":["Never hand-wrap with torch's fully_shard if you want transformers' distributed save; use distributed_config.","Check model._device_mesh is not None before saving.","Build EMA/eval copies only after establishing the distributed save path."],"tags":["fsdp","distributed","device-mesh","initialization"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}