hiyouga/LlamaFactory · error · ValueError

world_size ({helper.get_world_size()}) must be divisible by

Error message

world_size ({helper.get_world_size()}) must be divisible by mp_replicate_size ({self.mp_replicate_size}).

What it means

Raised while building the v1 `DistributedConfig` in `__post_init__`: in a distributed run the world size (total ranks from `torchrun`) must be evenly divisible by `mp_replicate_size` so that `mp_shard_size = world_size // mp_replicate_size` is a whole number. This sizes the model-parallel device mesh (replicate x shard).

Source

Thrown at src/llamafactory/v1/accelerator/interface.py:76

@dataclass
class DistributedStrategy:
    """Distributed strategy."""

    mp_replicate_size: int = 1
    """Model parallel replicate size, default to 1."""
    mp_shard_size: int | None = None
    """Model parallel shard size, default to world_size // mp_replicate_size."""
    dp_size: int | None = None
    """Data parallel size, default to world_size // cp_size."""
    cp_size: int = 1
    """Context parallel size, default to 1."""

    def __post_init__(self) -> None:
        if not helper.is_distributed():
            self.mp_shard_size = 1
        elif self.mp_shard_size is None:
            if helper.get_world_size() % self.mp_replicate_size != 0:
                raise ValueError(
                    f"world_size ({helper.get_world_size()}) must be divisible by "
                    f"mp_replicate_size ({self.mp_replicate_size})."
                )
            self.mp_shard_size = helper.get_world_size() // self.mp_replicate_size
        elif self.mp_replicate_size * self.mp_shard_size != helper.get_world_size():
            raise ValueError(
                f"mp_replicate_size * mp_shard_size must equal to world_size, "
                f"got {self.mp_replicate_size} * {self.mp_shard_size} != {helper.get_world_size()}."
            )

        if not helper.is_distributed():
            self.dp_size = 1
        elif self.dp_size is None:
            if helper.get_world_size() % self.cp_size != 0:
                raise ValueError(
                    f"world_size ({helper.get_world_size()}) must be divisible by cp_size ({self.cp_size})."
                )
            self.dp_size = helper.get_world_size() // self.cp_size

View on GitHub (pinned to f28afaf635)

Solutions

  1. Set `mp_replicate_size` to a divisor of your total world size (e.g. 1, 2, 4 for 8 ranks)
  2. Or change the launcher so world size matches, e.g. `torchrun --nproc_per_node=8` when `mp_replicate_size: 4`
  3. Double-check multi-node setups: world_size = nproc_per_node * num_nodes; recompute divisibility before launch

Example fix

# before: 4 GPUs, replicate over 3
# torchrun --nproc_per_node=4 train.py dist.mp_replicate_size=3

# after
torchrun --nproc_per_node=4 train.py dist.mp_replicate_size=2  # 4 % 2 == 0
Defensive patterns

Strategy: validation

Validate before calling

def check_mesh(world_size: int, mp_replicate_size: int) -> None:
    if world_size % mp_replicate_size != 0:
        raise SystemExit(
            f"mp_replicate_size={mp_replicate_size} must divide world_size={world_size}"
        )

# call with the launcher before trainer construction:
# check_mesh(int(os.environ["WORLD_SIZE"]), args.mp_replicate_size)

Try / catch

try:
    cfg = DistributedConfig(mp_replicate_size=rep)
except ValueError as e:
    if "divisible by" in str(e):
        # pick the largest divisor of world_size <= rep
        rep = max(d for d in range(1, rep + 1) if world_size % d == 0)
        cfg = DistributedConfig(mp_replicate_size=rep)
    else:
        raise

Prevention

When it happens

Trigger: Launching with `torchrun --nproc_per_node N` (or a Ray/multi-node equivalent) where N (times node count) is not a multiple of the configured `mp_replicate_size`, while `mp_shard_size` is left unset.

Common situations: Setting `mp_replicate_size: 3` on 4 GPUs; forgetting that a 2-node x 4-GPU job has world_size 8, not 4; copying a config tuned for 8 GPUs onto a single 4-GPU machine.

Related errors

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/bdfcc0da4ecb5cca. Report an issue: GitHub.