huggingface/transformers · error · ValueError

Saving an FSDP-wrapped model requires torch.distributed to b

Error message

Saving an FSDP-wrapped model requires torch.distributed to be initialized. Call save_pretrained from every rank after init_process_group.

What it means

When fsdp_size > 1 and you save without distributed_checkpoint (the gathered path), transformers calls torch.distributed collectives (gather_full_state_dict) which require an initialized process group. If torch.distributed.is_initialized() is False — e.g. single-process scripts or saving outside a torchrun context — the save raises ValueError instead of failing inside a NCCL collective with an opaque error.

Source

Thrown at src/transformers/distributed/mixin.py:277

        model_to_save,
        state_dict: dict,
        distributed_config: DistributedConfig | None,
        *,
        save_on_this_rank: bool = True,
    ) -> dict:
        """Gather TP- or FSDP-sharded weights to full CPU tensors for checkpoint writing."""
        if distributed_config is None:
            return state_dict

        if distributed_config.tp_size > 1:
            state_dict = gather_state_dict_for_save(state_dict, self._tp_plan, self._device_mesh, self._tp_size)
            if not save_on_this_rank:
                state_dict = {}
            return state_dict

        if distributed_config.fsdp_size > 1:
            if not _is_torch_distributed_initialized():
                raise ValueError(
                    "Saving an FSDP-wrapped model requires torch.distributed to be initialized. "
                    "Call save_pretrained from every rank after init_process_group."
                )
            return gather_full_state_dict(model_to_save)

        return state_dict

    def barrier_after_gathered_checkpoint_save(self, distributed_config: DistributedConfig | None) -> None:
        """Barrier so non-writer ranks wait for rank 0 to finish gathered checkpoint writes."""
        if distributed_config is None:
            return
        if distributed_config.tp_size > 1 or distributed_config.fsdp_size > 1:
            _distributed_barrier()

View on GitHub (pinned to a597f97485)

Solutions

Launch the script with torchrun (or set RANK/WORLD_SIZE/MASTER_ADDR/MASTER_PORT and call init_process_group) so all ranks save together.
Ensure save_pretrained is called on every rank, not just rank 0 — non-writer ranks participate in the gather.
If the process group was torn down, either re-init it before saving or save from a non-distributed context with fsdp_size=1.

Example fix

# before
python train.py  # world_size=1, fsdp gather fails

# after
torchrun --nproc_per_node=4 train.py  # and call model.save_pretrained(...) on ALL ranks

Defensive patterns

Strategy: validation

Validate before calling

import torch.distributed as dist

def assert_dist_for_fsdp_save() -> None:
    if not dist.is_initialized():
        raise RuntimeError(
            "Launch with torchrun and call init_process_group before save_pretrained on FSDP models"
        )

Try / catch

try:
    state = model.gathered_state_dict_for_save(cfg)  # or save_pretrained without distributed_checkpoint
except ValueError as e:
    if "torch.distributed to be initialized" in str(e):
        torch.distributed.init_process_group(backend="nccl")
        state = model.gathered_state_dict_for_save(cfg)
    else:
        raise

Prevention

Call save_pretrained on every rank, in the same iteration.
Never save after destroy_process_group().
Gate dev/debug single-process runs on fsdp_size=1.

When it happens

Trigger: Calling model.save_pretrained(...) in a script that never called torch.distributed.init_process_group (e.g. ran with plain python instead of torchrun); saving after destroy_process_group(); saving in a subprocess without the rendezvous env vars.

Common situations: Debugging in a notebook; post-training export scripts that reload a config with fsdp_size>1 but run single-process; calling destroy_process_group() for cleanup then attempting one more save.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/1376808732733823. Report an issue: GitHub.