What causes "FSDP2 requires `torch>=2.7` (distributed checkpoint save/load)." in huggingface/transformers?

huggingface/transformers · error · OSError

FSDP2 requires `torch>=2.7` (distributed checkpoint save/loa

Error message

FSDP2 requires `torch>=2.7` (distributed checkpoint save/load).

What it means

initialize_fully_sharded_data_parallelism guards FSDP2 setup with a torch version check: fully_shard itself needs torch>=2.6, but transformers' FSDP flow also depends on DCP with HuggingFaceStorageWriter for checkpoint save/load, which only ships in torch>=2.7. With fsdp_size>1 on older torch it raises OSError before any wrapping happens.

Source

Thrown at src/transformers/distributed/utils.py:120

    Passing `device_ids` is required when the process group was initialized without a
    `device_id`; with it, the call is a no-op compared to plain `barrier()`. Safe to call
    when torch.distributed has not been initialized — returns immediately.
    """
    if not _is_torch_distributed_initialized():
        return
    device_type = torch._C._get_accelerator().type
    if device_type != "cpu":
        torch.distributed.barrier(device_ids=[getattr(torch, device_type).current_device()])
    else:
        torch.distributed.barrier()


def initialize_fully_sharded_data_parallelism(distributed_config: DistributedConfig):
    # `fully_shard` itself only needs torch>=2.6, but distributed checkpoint save/load
    # (DCP + HuggingFaceStorageWriter) needs 2.7, so that is the effective requirement.
    if distributed_config.fsdp_size > 1 and not is_torch_greater_or_equal("2.7"):
        raise OSError("FSDP2 requires `torch>=2.7` (distributed checkpoint save/load).")

    device_type = torch._C._get_accelerator().type

    if device_type != "cpu":
        local_rank = int(os.environ.get("LOCAL_RANK", 0))
        getattr(torch, device_type).set_device(local_rank)
        device_map = torch.device(device_type, local_rank)
    else:
        device_map = torch.device(device_type)

    fsdp_size = distributed_config.fsdp_size

    dims, names = [], []
    if fsdp_size > 1:
        dims.append(fsdp_size)
        names.append("fsdp")

    # Build the N-dimensional device mesh

View on GitHub (pinned to a597f97485)

Solutions

pip install -U 'torch>=2.7'.
If stuck on older torch, use torch's legacy FSDP (fully_shard unavailable) via your own wrapping, or train without FSDP (fsdp_size=1).
Verify in CI with a pinned check: torch version >= 2.7 before requesting FSDP2.

Example fix

# before
torch==2.6.0; distributed_config={"fsdp_size": 4}  # OSError

# after
torch>=2.7; distributed_config={"fsdp_size": 4}

Defensive patterns

Strategy: validation

Validate before calling

from transformers.utils import is_torch_greater_or_equal

def assert_torch_for_fsdp2() -> None:
    if not is_torch_greater_or_equal("2.7"):
        raise RuntimeError("FSDP2 path needs torch>=2.7 (DCP/HuggingFaceStorageWriter); upgrade torch or set fsdp_size=1")

Try / catch

try:
    Model.from_pretrained(model_id, distributed_config={"fsdp_size": 4})
except OSError as e:
    if "FSDP2 requires" in str(e):
        cfg.pop("fsdp_size")  # degrade to unsharded for smoke tests
        Model.from_pretrained(model_id)
    else:
        raise

Prevention

Pin torch>=2.7 whenever distributed_config contains fsdp_size>1.
Run the version assertion in CI before long training jobs.
Do not mix a new transformers with an old torch for FSDP workflows.

When it happens

Trigger: AutoModelForCausalLM.from_pretrained(..., distributed_config={'fsdp_size': N>1}) in an environment with torch < 2.7.

Common situations: CUDA-base images pinned to torch 2.5/2.6; upgrading transformers for the new FSDP2 support without upgrading torch; corporate environments with slow torch rollouts.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/23d45bed4bc34c70. Report an issue: GitHub.