{"record":{"id":"7f11f03914197aea","repo":"huggingface/transformers","slug":"distributed-checkpointing-requires-torch-2-7","errorCode":null,"errorMessage":"Distributed checkpointing requires `torch>=2.7`.","messagePattern":"Distributed checkpointing requires `torch>=2\\.7`\\.","errorType":"exception","errorClass":"OSError","httpStatus":null,"severity":"error","filePath":"src/transformers/distributed/utils.py","lineNumber":153,"sourceCode":"        dims.append(fsdp_size)\n        names.append(\"fsdp\")\n\n    # Build the N-dimensional device mesh\n    mesh = torch.distributed.init_device_mesh(device_type, tuple(dims), mesh_dim_names=tuple(names))\n    # If N > 1, create a flattened sub-mesh so all-reduces across the world mesh ae done in one collective\n    if len(dims) > 1:\n        mesh._flatten(\"_\".join(names))\n\n    return device_map, mesh\n\n\ndef gather_full_state_dict(model) -> dict[str, torch.Tensor]:\n    \"\"\"Gather FSDP-sharded params to full plain CPU tensors.\n\n    Only rank 0 accumulates the result; other ranks return ``{}``.\n    \"\"\"\n    if not is_torch_greater_or_equal(\"2.7\"):\n        raise OSError(\"Distributed checkpointing requires `torch>=2.7`.\")\n\n    # Import here because otherwise it emits a warning every time it's imported on some hardware - this keeps the warning from\n    # being emitted if the function is not used\n    from torch.distributed.checkpoint.state_dict import StateDictOptions, get_model_state_dict\n\n    options = StateDictOptions(full_state_dict=True, cpu_offload=True)\n    full_state_dict = get_model_state_dict(model, options=options)\n    if _get_torch_distributed_rank() == 0:\n        return full_state_dict\n    return {}\n\n\ndef save_model_checkpoint_distributed(model, checkpoint_dir: str) -> None:\n    \"\"\"Save model parameters as standard HF-format sharded safetensors using\n    DCP + HuggingFaceStorageWriter with consolidation enabled.\n\n    Every rank first writes its own shard in parallel under\n    `<checkpoint_dir>/sharded/`, then a consolidation pass reads those shards","sourceCodeStart":135,"sourceCodeEnd":171,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/distributed/utils.py#L135-L171","documentation":"gather_full_state_dict uses torch.distributed.checkpoint.state_dict.get_model_state_dict with full_state_dict/cpu_offload options, an API surface guaranteed only from torch 2.7. On older versions it raises OSError immediately rather than failing with an AttributeError deep inside torch.","triggerScenarios":"Calling model.save_pretrained(...) on an FSDP-wrapped model (fsdp_size>1, gathered path) with torch < 2.7 installed.","commonSituations":"Same class of issue as the other version guards: environments pinned below torch 2.7 attempting the transformers distributed checkpoint flow.","solutions":["Upgrade torch to >= 2.7.","On older torch, unwrap FSDP manually and save the full state dict with torch's own state_dict utilities.","Keep torch and transformers versions in lockstep in your environment files."],"exampleFix":"# before\npip install torch==2.6.0  # then save_pretrained raises OSError\n\n# after\npip install 'torch>=2.7'","handlingStrategy":"validation","validationCode":"from transformers.utils import is_torch_greater_or_equal\nassert is_torch_greater_or_equal(\"2.7\"), \"gather_full_state_dict needs torch>=2.7\"","typeGuard":null,"tryCatchPattern":"try:\n    full = gather_full_state_dict(model)\nexcept OSError as e:\n    if \"torch>=2.7\" in str(e):\n        full = {k: v.cpu() for k, v in model.state_dict().items()}  # manual fallback only for unwrapped models\n    else:\n        raise","preventionTips":["Version-check once at startup and fail fast.","Keep the torch floor declared in requirements.","Centralize version guards instead of scattering try/except."],"tags":["torch-version","fsdp","checkpointing","distributed"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}