huggingface/transformers · error · OSError

save_pretrained(..., distributed_checkpoint=True) requires…

Error message

save_pretrained(..., distributed_checkpoint=True) requires torch>=2.7.

What it means

The distributed-checkpoint save path (DCP + HuggingFaceStorageWriter) used by save_pretrained(..., distributed_checkpoint=True) relies on APIs that only exist in torch 2.7 and newer. On older torch the save raises OSError before touching the disk, so no partial checkpoint is written.

Solutions

Upgrade torch: pip install -U 'torch>=2.7'.
If torch cannot be upgraded, save with the regular gathered path (distributed_checkpoint=False) which full-state-dict gathers to rank 0.
Pin transformers/torch together in requirements to avoid silent version drift.

Example fix

# before
torch==2.6.0  # requirements.txt, save_pretrained(..., distributed_checkpoint=True) raises OSError

# after
torch>=2.7.0  # requirements.txt

Defensive patterns

Strategy: validation

Validate before calling

from transformers.utils import is_torch_greater_or_equal

def assert_torch_for_distributed_save() -> None:
    if not is_torch_greater_or_equal("2.7"):
        raise RuntimeError("distributed_checkpoint=True needs torch>=2.7; upgrade or save gathered instead")

Try / catch

try:
    model.save_pretrained(out, distributed_checkpoint=True)
except OSError as e:
    if "requires torch>=2.7" in str(e):
        model.save_pretrained(out)  # fallback: gathered, rank-0 save
    else:
        raise

Prevention

Assert the torch version at trainer startup, not at save time after hours of training.
Pin 'torch>=2.7' in the environment spec when using distributed checkpoints.
Wrap the save with a fallback to the non-distributed path if you support mixed fleets.

When it happens

Trigger: Calling Model.save_pretrained(..., distributed_checkpoint=True) (or the mixin's save_pretrained_distributed) in an environment where importlib metadata reports torch < 2.7.

Common situations: Training environments pinned to torch 2.4-2.6 (common for older CUDA images); CI images lagging behind; upgrading transformers without upgrading torch.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/f67c612a8afca0aa. Report an issue: GitHub.

Appendix: source

Thrown at src/transformers/distributed/mixin.py:233

            save_on_this_rank = save_on_this_rank and _get_torch_distributed_rank() == 0
        return save_on_this_rank

    def save_distributed_checkpoint(
        self,
        model_to_save,
        save_directory: str | os.PathLike,
        *,
        push_to_hub: bool = False,
        save_on_this_rank: bool = True,
        repo_id: str | None = None,
        files_timestamps: dict | None = None,
        commit_message: str | None = None,
        token: str | bool | None = None,
        create_pr: bool = False,
    ) -> None:
        """Save an FSDP-wrapped model via DCP and optionally push to the Hub."""
        if not is_torch_greater_or_equal("2.7"):
            raise OSError("save_pretrained(..., distributed_checkpoint=True) requires torch>=2.7.")
        if not is_fsdp_managed_module(model_to_save):
            raise ValueError(
                "save_pretrained(..., distributed_checkpoint=True) is only supported for FSDP-wrapped models."
            )
        if getattr(model_to_save, "_device_mesh", None) is None:
            raise ValueError(
                "save_pretrained(..., distributed_checkpoint=True) requires the model to have been "
                "initialized with a distributed_config (_device_mesh is None)."
            )
        save_model_checkpoint_distributed(model_to_save, save_directory)

        if push_to_hub and save_on_this_rank:
            model_card = create_and_tag_model_card(repo_id, self.model_tags, token=token)
            model_card.save(os.path.join(save_directory, "README.md"))
            self._upload_modified_files(
                save_directory,
                repo_id,
                files_timestamps,

View on GitHub (pinned to a597f97485)