{"record":{"id":"1376808732733823","repo":"huggingface/transformers","slug":"saving-an-fsdp-wrapped-model-requires-torch-distri","errorCode":null,"errorMessage":"Saving an FSDP-wrapped model requires torch.distributed to be initialized. Call save_pretrained from every rank after init_process_group.","messagePattern":"Saving an FSDP-wrapped model requires torch\\.distributed to be initialized\\. Call save_pretrained from every rank after init_process_group\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/distributed/mixin.py","lineNumber":277,"sourceCode":"        model_to_save,\n        state_dict: dict,\n        distributed_config: DistributedConfig | None,\n        *,\n        save_on_this_rank: bool = True,\n    ) -> dict:\n        \"\"\"Gather TP- or FSDP-sharded weights to full CPU tensors for checkpoint writing.\"\"\"\n        if distributed_config is None:\n            return state_dict\n\n        if distributed_config.tp_size > 1:\n            state_dict = gather_state_dict_for_save(state_dict, self._tp_plan, self._device_mesh, self._tp_size)\n            if not save_on_this_rank:\n                state_dict = {}\n            return state_dict\n\n        if distributed_config.fsdp_size > 1:\n            if not _is_torch_distributed_initialized():\n                raise ValueError(\n                    \"Saving an FSDP-wrapped model requires torch.distributed to be initialized. \"\n                    \"Call save_pretrained from every rank after init_process_group.\"\n                )\n            return gather_full_state_dict(model_to_save)\n\n        return state_dict\n\n    def barrier_after_gathered_checkpoint_save(self, distributed_config: DistributedConfig | None) -> None:\n        \"\"\"Barrier so non-writer ranks wait for rank 0 to finish gathered checkpoint writes.\"\"\"\n        if distributed_config is None:\n            return\n        if distributed_config.tp_size > 1 or distributed_config.fsdp_size > 1:\n            _distributed_barrier()\n","sourceCodeStart":259,"sourceCodeEnd":291,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/distributed/mixin.py#L259-L291","documentation":"When fsdp_size > 1 and you save without distributed_checkpoint (the gathered path), transformers calls torch.distributed collectives (gather_full_state_dict) which require an initialized process group. If torch.distributed.is_initialized() is False — e.g. single-process scripts or saving outside a torchrun context — the save raises ValueError instead of failing inside a NCCL collective with an opaque error.","triggerScenarios":"Calling model.save_pretrained(...) in a script that never called torch.distributed.init_process_group (e.g. ran with plain python instead of torchrun); saving after destroy_process_group(); saving in a subprocess without the rendezvous env vars.","commonSituations":"Debugging in a notebook; post-training export scripts that reload a config with fsdp_size>1 but run single-process; calling destroy_process_group() for cleanup then attempting one more save.","solutions":["Launch the script with torchrun (or set RANK/WORLD_SIZE/MASTER_ADDR/MASTER_PORT and call init_process_group) so all ranks save together.","Ensure save_pretrained is called on every rank, not just rank 0 — non-writer ranks participate in the gather.","If the process group was torn down, either re-init it before saving or save from a non-distributed context with fsdp_size=1."],"exampleFix":"# before\npython train.py  # world_size=1, fsdp gather fails\n\n# after\ntorchrun --nproc_per_node=4 train.py  # and call model.save_pretrained(...) on ALL ranks","handlingStrategy":"validation","validationCode":"import torch.distributed as dist\n\ndef assert_dist_for_fsdp_save() -> None:\n    if not dist.is_initialized():\n        raise RuntimeError(\n            \"Launch with torchrun and call init_process_group before save_pretrained on FSDP models\"\n        )","typeGuard":null,"tryCatchPattern":"try:\n    state = model.gathered_state_dict_for_save(cfg)  # or save_pretrained without distributed_checkpoint\nexcept ValueError as e:\n    if \"torch.distributed to be initialized\" in str(e):\n        torch.distributed.init_process_group(backend=\"nccl\")\n        state = model.gathered_state_dict_for_save(cfg)\n    else:\n        raise","preventionTips":["Call save_pretrained on every rank, in the same iteration.","Never save after destroy_process_group().","Gate dev/debug single-process runs on fsdp_size=1."],"tags":["fsdp","distributed","checkpointing","process-group"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}