What causes "Distributed checkpointing requires `torch>=2.7`." in huggingface/transformers?

huggingface/transformers · error · OSError

Distributed checkpointing requires `torch>=2.7`.

Error message

Distributed checkpointing requires `torch>=2.7`.

What it means

gather_full_state_dict uses torch.distributed.checkpoint.state_dict.get_model_state_dict with full_state_dict/cpu_offload options, an API surface guaranteed only from torch 2.7. On older versions it raises OSError immediately rather than failing with an AttributeError deep inside torch.

Solutions

Upgrade torch to >= 2.7.
On older torch, unwrap FSDP manually and save the full state dict with torch's own state_dict utilities.
Keep torch and transformers versions in lockstep in your environment files.

Example fix

# before
pip install torch==2.6.0  # then save_pretrained raises OSError

# after
pip install 'torch>=2.7'

Defensive patterns

Strategy: validation

Validate before calling

from transformers.utils import is_torch_greater_or_equal
assert is_torch_greater_or_equal("2.7"), "gather_full_state_dict needs torch>=2.7"

Try / catch

try:
    full = gather_full_state_dict(model)
except OSError as e:
    if "torch>=2.7" in str(e):
        full = {k: v.cpu() for k, v in model.state_dict().items()}  # manual fallback only for unwrapped models
    else:
        raise

Prevention

Version-check once at startup and fail fast.
Keep the torch floor declared in requirements.
Centralize version guards instead of scattering try/except.

When it happens

Trigger: Calling model.save_pretrained(...) on an FSDP-wrapped model (fsdp_size>1, gathered path) with torch < 2.7 installed.

Common situations: Same class of issue as the other version guards: environments pinned below torch 2.7 attempting the transformers distributed checkpoint flow.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/7f11f03914197aea. Report an issue: GitHub.

Appendix: source

Thrown at src/transformers/distributed/utils.py:153

        dims.append(fsdp_size)
        names.append("fsdp")

    # Build the N-dimensional device mesh
    mesh = torch.distributed.init_device_mesh(device_type, tuple(dims), mesh_dim_names=tuple(names))
    # If N > 1, create a flattened sub-mesh so all-reduces across the world mesh ae done in one collective
    if len(dims) > 1:
        mesh._flatten("_".join(names))

    return device_map, mesh


def gather_full_state_dict(model) -> dict[str, torch.Tensor]:
    """Gather FSDP-sharded params to full plain CPU tensors.

    Only rank 0 accumulates the result; other ranks return ``{}``.
    """
    if not is_torch_greater_or_equal("2.7"):
        raise OSError("Distributed checkpointing requires `torch>=2.7`.")

    # Import here because otherwise it emits a warning every time it's imported on some hardware - this keeps the warning from
    # being emitted if the function is not used
    from torch.distributed.checkpoint.state_dict import StateDictOptions, get_model_state_dict

    options = StateDictOptions(full_state_dict=True, cpu_offload=True)
    full_state_dict = get_model_state_dict(model, options=options)
    if _get_torch_distributed_rank() == 0:
        return full_state_dict
    return {}


def save_model_checkpoint_distributed(model, checkpoint_dir: str) -> None:
    """Save model parameters as standard HF-format sharded safetensors using
    DCP + HuggingFaceStorageWriter with consolidation enabled.

    Every rank first writes its own shard in parallel under
    `<checkpoint_dir>/sharded/`, then a consolidation pass reads those shards

View on GitHub (pinned to a597f97485)