{"record":{"id":"f4723252cf5d3e89","repo":"huggingface/transformers","slug":"tp-size-distributed-config-tp-size-fsdp-size","errorCode":null,"errorMessage":"tp_size ({distributed_config.tp_size}) * fsdp_size ({distributed_config.fsdp_size}) is not equal to world_size ({world_size})","messagePattern":"tp_size \\((.+?)\\) \\* fsdp_size \\((.+?)\\) is not equal to world_size \\((.+?)\\)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"src/transformers/distributed/mixin.py","lineNumber":168,"sourceCode":"    @classmethod\n    def prepare_distribute_model(\n        cls,\n        distributed_config: DistributedConfig | dict | None,\n        *,\n        device_mesh=None,\n        device_map=None,\n    ) -> tuple[DistributedConfig | None, object, object]:\n        if distributed_config is None:\n            return None, device_map, device_mesh\n\n        if isinstance(distributed_config, dict):\n            distributed_config = DistributedConfig.from_dict(distributed_config)\n\n        if distributed_config.tp_size > 1 or distributed_config.fsdp_size > 1:\n            _ensure_torch_distributed()\n            world_size = _get_torch_distributed_world_size()\n            if distributed_config.tp_size * distributed_config.fsdp_size != world_size:\n                raise RuntimeError(\n                    f\"tp_size ({distributed_config.tp_size}) * fsdp_size ({distributed_config.fsdp_size}) \"\n                    f\"is not equal to world_size ({world_size})\"\n                )\n\n        if distributed_config.tp_size > 1:\n            if distributed_config.tp_plan is None:\n                distributed_config.tp_plan = \"auto\"\n            device_map, device_mesh = initialize_tensor_parallelism(\n                distributed_config.tp_plan,\n                tp_size=distributed_config.tp_size,\n                device_mesh=device_mesh,\n                device_map=device_map,\n            )\n        elif distributed_config.fsdp_size > 1:\n            device_map, device_mesh = initialize_fully_sharded_data_parallelism(distributed_config)\n\n        return distributed_config, device_map, device_mesh\n","sourceCodeStart":150,"sourceCodeEnd":186,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/distributed/mixin.py#L150-L186","documentation":"prepare_distribute_model checks that the product of distributed_config.tp_size and distributed_config.fsdp_size equals the torch.distributed world size before setting up TP/FSDP. Since TP shards layers and FSDP shards parameters over the same process group, the replication degree of the world must be exactly covered by these two factors; otherwise the mesh cannot be built and a RuntimeError is raised.","triggerScenarios":"Launching torchrun --nproc_per_node=8 but setting distributed_config={'tp_size': 4, 'fsdp_size': 4} (16 != 8); or launching single-process (world_size=1) while requesting tp_size=2; or forgetting to scale sizes after changing the number of GPUs.","commonSituations":"Changing GPU count without updating tp_size/fsdp_size in the config; copying a config tuned for multi-node to a single node; running in a notebook where torch.distributed was initialized with a stale world size.","solutions":["Make tp_size * fsdp_size equal the number of launched ranks, e.g. torchrun --nproc_per_node=8 with tp_size=2, fsdp_size=4.","If you only want FSDP, set tp_size=1 and fsdp_size=world_size (and vice versa for pure TP).","If the model is small or single-GPU, drop the distributed_config entirely."],"exampleFix":"# before (launched with 8 ranks)\ndistributed_config = {\"tp_size\": 4, \"fsdp_size\": 4}\n\n# after\ndistributed_config = {\"tp_size\": 2, \"fsdp_size\": 4}  # 2*4 == 8","handlingStrategy":"validation","validationCode":"import torch.distributed as dist\n\ndef check_sizes(tp_size: int, fsdp_size: int) -> None:\n    if tp_size > 1 or fsdp_size > 1:\n        if not dist.is_initialized():\n            raise RuntimeError(\"initialize torch.distributed (launch with torchrun) first\")\n        if tp_size * fsdp_size != dist.get_world_size():\n            raise ValueError(\n                f\"tp_size*fsdp_size={tp_size * fsdp_size} != world_size={dist.get_world_size()}; \"\n                f\"launch torchrun with --nproc_per_node={tp_size * fsdp_size}\"\n            )","typeGuard":null,"tryCatchPattern":"try:\n    Model.from_pretrained(model_id, distributed_config=cfg)\nexcept RuntimeError as e:\n    if \"is not equal to world_size\" in str(e):\n        sys.exit(\"Fix --nproc_per_node or tp_size/fsdp_size: \" + str(e))\n    raise","preventionTips":["Derive tp_size/fsdp_size from dist.get_world_size() at runtime instead of hardcoding.","Fail fast in your trainer main() before model load.","Make world size part of the config, not of the launch script folklore."],"tags":["distributed","fsdp","tensor-parallel","world-size","config"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}