{"record":{"id":"23d45bed4bc34c70","repo":"huggingface/transformers","slug":"fsdp2-requires-torch-2-7-distributed-checkpoin","errorCode":null,"errorMessage":"FSDP2 requires `torch>=2.7` (distributed checkpoint save/load).","messagePattern":"FSDP2 requires `torch>=2\\.7` \\(distributed checkpoint save/load\\)\\.","errorType":"exception","errorClass":"OSError","httpStatus":null,"severity":"error","filePath":"src/transformers/distributed/utils.py","lineNumber":120,"sourceCode":"\n    Passing `device_ids` is required when the process group was initialized without a\n    `device_id`; with it, the call is a no-op compared to plain `barrier()`. Safe to call\n    when torch.distributed has not been initialized — returns immediately.\n    \"\"\"\n    if not _is_torch_distributed_initialized():\n        return\n    device_type = torch._C._get_accelerator().type\n    if device_type != \"cpu\":\n        torch.distributed.barrier(device_ids=[getattr(torch, device_type).current_device()])\n    else:\n        torch.distributed.barrier()\n\n\ndef initialize_fully_sharded_data_parallelism(distributed_config: DistributedConfig):\n    # `fully_shard` itself only needs torch>=2.6, but distributed checkpoint save/load\n    # (DCP + HuggingFaceStorageWriter) needs 2.7, so that is the effective requirement.\n    if distributed_config.fsdp_size > 1 and not is_torch_greater_or_equal(\"2.7\"):\n        raise OSError(\"FSDP2 requires `torch>=2.7` (distributed checkpoint save/load).\")\n\n    device_type = torch._C._get_accelerator().type\n\n    if device_type != \"cpu\":\n        local_rank = int(os.environ.get(\"LOCAL_RANK\", 0))\n        getattr(torch, device_type).set_device(local_rank)\n        device_map = torch.device(device_type, local_rank)\n    else:\n        device_map = torch.device(device_type)\n\n    fsdp_size = distributed_config.fsdp_size\n\n    dims, names = [], []\n    if fsdp_size > 1:\n        dims.append(fsdp_size)\n        names.append(\"fsdp\")\n\n    # Build the N-dimensional device mesh","sourceCodeStart":102,"sourceCodeEnd":138,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/distributed/utils.py#L102-L138","documentation":"initialize_fully_sharded_data_parallelism guards FSDP2 setup with a torch version check: fully_shard itself needs torch>=2.6, but transformers' FSDP flow also depends on DCP with HuggingFaceStorageWriter for checkpoint save/load, which only ships in torch>=2.7. With fsdp_size>1 on older torch it raises OSError before any wrapping happens.","triggerScenarios":"AutoModelForCausalLM.from_pretrained(..., distributed_config={'fsdp_size': N>1}) in an environment with torch < 2.7.","commonSituations":"CUDA-base images pinned to torch 2.5/2.6; upgrading transformers for the new FSDP2 support without upgrading torch; corporate environments with slow torch rollouts.","solutions":["pip install -U 'torch>=2.7'.","If stuck on older torch, use torch's legacy FSDP (fully_shard unavailable) via your own wrapping, or train without FSDP (fsdp_size=1).","Verify in CI with a pinned check: torch version >= 2.7 before requesting FSDP2."],"exampleFix":"# before\ntorch==2.6.0; distributed_config={\"fsdp_size\": 4}  # OSError\n\n# after\ntorch>=2.7; distributed_config={\"fsdp_size\": 4}","handlingStrategy":"validation","validationCode":"from transformers.utils import is_torch_greater_or_equal\n\ndef assert_torch_for_fsdp2() -> None:\n    if not is_torch_greater_or_equal(\"2.7\"):\n        raise RuntimeError(\"FSDP2 path needs torch>=2.7 (DCP/HuggingFaceStorageWriter); upgrade torch or set fsdp_size=1\")","typeGuard":null,"tryCatchPattern":"try:\n    Model.from_pretrained(model_id, distributed_config={\"fsdp_size\": 4})\nexcept OSError as e:\n    if \"FSDP2 requires\" in str(e):\n        cfg.pop(\"fsdp_size\")  # degrade to unsharded for smoke tests\n        Model.from_pretrained(model_id)\n    else:\n        raise","preventionTips":["Pin torch>=2.7 whenever distributed_config contains fsdp_size>1.","Run the version assertion in CI before long training jobs.","Do not mix a new transformers with an old torch for FSDP workflows."],"tags":["fsdp","torch-version","distributed","environment"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}