hiyouga/LlamaFactory · error · ValueError

mp_replicate_size * mp_shard_size must equal to world_size,

Error message

mp_replicate_size * mp_shard_size must equal to world_size, got {self.mp_replicate_size} * {self.mp_shard_size} != {helper.get_world_size()}.

What it means

Raised in `DistributedConfig.__post_init__` when BOTH `mp_replicate_size` and `mp_shard_size` are explicitly provided in a distributed run but their product does not equal world size. The mesh must exactly cover all ranks, so `mp_replicate_size * mp_shard_size == world_size` is an invariant.

Source

Thrown at src/llamafactory/v1/accelerator/interface.py:82

    mp_shard_size: int | None = None
    """Model parallel shard size, default to world_size // mp_replicate_size."""
    dp_size: int | None = None
    """Data parallel size, default to world_size // cp_size."""
    cp_size: int = 1
    """Context parallel size, default to 1."""

    def __post_init__(self) -> None:
        if not helper.is_distributed():
            self.mp_shard_size = 1
        elif self.mp_shard_size is None:
            if helper.get_world_size() % self.mp_replicate_size != 0:
                raise ValueError(
                    f"world_size ({helper.get_world_size()}) must be divisible by "
                    f"mp_replicate_size ({self.mp_replicate_size})."
                )
            self.mp_shard_size = helper.get_world_size() // self.mp_replicate_size
        elif self.mp_replicate_size * self.mp_shard_size != helper.get_world_size():
            raise ValueError(
                f"mp_replicate_size * mp_shard_size must equal to world_size, "
                f"got {self.mp_replicate_size} * {self.mp_shard_size} != {helper.get_world_size()}."
            )

        if not helper.is_distributed():
            self.dp_size = 1
        elif self.dp_size is None:
            if helper.get_world_size() % self.cp_size != 0:
                raise ValueError(
                    f"world_size ({helper.get_world_size()}) must be divisible by cp_size ({self.cp_size})."
                )
            self.dp_size = helper.get_world_size() // self.cp_size
        elif self.dp_size * self.cp_size != helper.get_world_size():
            raise ValueError(
                f"dp_size * cp_size must equal to world_size, "
                f"got {self.dp_size} * {self.cp_size} != {helper.get_world_size()}."
            )

View on GitHub (pinned to f28afaf635)

Solutions

Adjust one of the two sizes so the product equals world size (e.g. for 4 ranks use 2x2 or 1x4)
Or drop `mp_shard_size` from the config and let it be derived as `world_size // mp_replicate_size` (which triggers the friendlier divisibility check instead)
Verify world size first: `echo $WORLD_SIZE` or check the launcher's process count

Example fix

# before: single node, 4 GPUs
dist:
  mp_replicate_size: 2
  mp_shard_size: 4   # 2*4=8 != 4

# after
dist:
  mp_replicate_size: 2
  mp_shard_size: 2   # 2*2=4 == world_size

Defensive patterns

Strategy: validation

Validate before calling

def validate_mesh(world_size: int, mp_replicate: int, mp_shard: int) -> None:
    if mp_replicate * mp_shard != world_size:
        raise SystemExit(
            f"mp mesh {mp_replicate}x{mp_shard}={mp_replicate * mp_shard} != world_size {world_size}"
        )

Prevention

Set only mp_replicate_size and let mp_shard_size be derived
Validate the product in the job template before submitting to a scheduler
Regenerate configs on GPU-count changes instead of editing by hand

When it happens

Trigger: Explicitly configuring both `mp_replicate_size` and `mp_shard_size` in the v1 dist config while launching with a different number of processes, e.g. 2x4 mesh on 8 ranks launched with `--nproc_per_node=4` on one node.

Common situations: Reusing a multi-node config on fewer nodes; resizing GPU count without updating the mesh; assuming the trainer will silently clamp extra ranks.

Related errors

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/a28efda47523dc1c. Report an issue: GitHub.