huggingface/transformers · error · RuntimeError

tp_size ({distributed_config.tp_size}) * fsdp_size ({distrib

Error message

tp_size ({distributed_config.tp_size}) * fsdp_size ({distributed_config.fsdp_size}) is not equal to world_size ({world_size})

What it means

prepare_distribute_model checks that the product of distributed_config.tp_size and distributed_config.fsdp_size equals the torch.distributed world size before setting up TP/FSDP. Since TP shards layers and FSDP shards parameters over the same process group, the replication degree of the world must be exactly covered by these two factors; otherwise the mesh cannot be built and a RuntimeError is raised.

Source

Thrown at src/transformers/distributed/mixin.py:168

    @classmethod
    def prepare_distribute_model(
        cls,
        distributed_config: DistributedConfig | dict | None,
        *,
        device_mesh=None,
        device_map=None,
    ) -> tuple[DistributedConfig | None, object, object]:
        if distributed_config is None:
            return None, device_map, device_mesh

        if isinstance(distributed_config, dict):
            distributed_config = DistributedConfig.from_dict(distributed_config)

        if distributed_config.tp_size > 1 or distributed_config.fsdp_size > 1:
            _ensure_torch_distributed()
            world_size = _get_torch_distributed_world_size()
            if distributed_config.tp_size * distributed_config.fsdp_size != world_size:
                raise RuntimeError(
                    f"tp_size ({distributed_config.tp_size}) * fsdp_size ({distributed_config.fsdp_size}) "
                    f"is not equal to world_size ({world_size})"
                )

        if distributed_config.tp_size > 1:
            if distributed_config.tp_plan is None:
                distributed_config.tp_plan = "auto"
            device_map, device_mesh = initialize_tensor_parallelism(
                distributed_config.tp_plan,
                tp_size=distributed_config.tp_size,
                device_mesh=device_mesh,
                device_map=device_map,
            )
        elif distributed_config.fsdp_size > 1:
            device_map, device_mesh = initialize_fully_sharded_data_parallelism(distributed_config)

        return distributed_config, device_map, device_mesh

View on GitHub (pinned to a597f97485)

Solutions

Make tp_size * fsdp_size equal the number of launched ranks, e.g. torchrun --nproc_per_node=8 with tp_size=2, fsdp_size=4.
If you only want FSDP, set tp_size=1 and fsdp_size=world_size (and vice versa for pure TP).
If the model is small or single-GPU, drop the distributed_config entirely.

Example fix

# before (launched with 8 ranks)
distributed_config = {"tp_size": 4, "fsdp_size": 4}

# after
distributed_config = {"tp_size": 2, "fsdp_size": 4}  # 2*4 == 8

Defensive patterns

Strategy: validation

Validate before calling

import torch.distributed as dist

def check_sizes(tp_size: int, fsdp_size: int) -> None:
    if tp_size > 1 or fsdp_size > 1:
        if not dist.is_initialized():
            raise RuntimeError("initialize torch.distributed (launch with torchrun) first")
        if tp_size * fsdp_size != dist.get_world_size():
            raise ValueError(
                f"tp_size*fsdp_size={tp_size * fsdp_size} != world_size={dist.get_world_size()}; "
                f"launch torchrun with --nproc_per_node={tp_size * fsdp_size}"
            )

Try / catch

try:
    Model.from_pretrained(model_id, distributed_config=cfg)
except RuntimeError as e:
    if "is not equal to world_size" in str(e):
        sys.exit("Fix --nproc_per_node or tp_size/fsdp_size: " + str(e))
    raise

Prevention

Derive tp_size/fsdp_size from dist.get_world_size() at runtime instead of hardcoding.
Fail fast in your trainer main() before model load.
Make world size part of the config, not of the launch script folklore.

When it happens

Trigger: Launching torchrun --nproc_per_node=8 but setting distributed_config={'tp_size': 4, 'fsdp_size': 4} (16 != 8); or launching single-process (world_size=1) while requesting tp_size=2; or forgetting to scale sizes after changing the number of GPUs.

Common situations: Changing GPU count without updating tp_size/fsdp_size in the config; copying a config tuned for multi-node to a single node; running in a notebook where torch.distributed was initialized with a stale world size.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/f4723252cf5d3e89. Report an issue: GitHub.