hiyouga/LlamaFactory · error · ValueError

world_size ({helper.get_world_size()}) must be divisible by

Error message

world_size ({helper.get_world_size()}) must be divisible by cp_size ({self.cp_size}).

What it means

Raised in `DistributedConfig.__post_init__` when `dp_size` is unset in a distributed run and world size is not divisible by `cp_size` (context parallel size). The data-parallel dimension is derived as `world_size // cp_size`, so context parallelism must partition the ranks evenly.

Source

Thrown at src/llamafactory/v1/accelerator/interface.py:91

            self.mp_shard_size = 1
        elif self.mp_shard_size is None:
            if helper.get_world_size() % self.mp_replicate_size != 0:
                raise ValueError(
                    f"world_size ({helper.get_world_size()}) must be divisible by "
                    f"mp_replicate_size ({self.mp_replicate_size})."
                )
            self.mp_shard_size = helper.get_world_size() // self.mp_replicate_size
        elif self.mp_replicate_size * self.mp_shard_size != helper.get_world_size():
            raise ValueError(
                f"mp_replicate_size * mp_shard_size must equal to world_size, "
                f"got {self.mp_replicate_size} * {self.mp_shard_size} != {helper.get_world_size()}."
            )

        if not helper.is_distributed():
            self.dp_size = 1
        elif self.dp_size is None:
            if helper.get_world_size() % self.cp_size != 0:
                raise ValueError(
                    f"world_size ({helper.get_world_size()}) must be divisible by cp_size ({self.cp_size})."
                )
            self.dp_size = helper.get_world_size() // self.cp_size
        elif self.dp_size * self.cp_size != helper.get_world_size():
            raise ValueError(
                f"dp_size * cp_size must equal to world_size, "
                f"got {self.dp_size} * {self.cp_size} != {helper.get_world_size()}."
            )

    @property
    def model_mesh_shape(self) -> tuple[int, int]:
        """Model parallel mesh shape."""
        return (self.mp_replicate_size, self.mp_shard_size)

    @property
    def model_mesh_dim_names(self) -> tuple[str, str]:
        """Model parallel mesh dimension names."""
        return (Dim.MP_REPLICATE.value, Dim.MP_SHARD.value)

View on GitHub (pinned to f28afaf635)

Solutions

Use a `cp_size` that divides world size evenly (1, 2, 4, 8 on 8 ranks)
If you need an unusual split, set `dp_size` explicitly so the product check applies instead
Remember `cp_size` cannot exceed world size; reduce it if you shrank the job

Example fix

# before: 8 ranks, cp=3
training:
  cp_size: 3   # 8 % 3 != 0

# after
training:
  cp_size: 4   # 8 % 4 == 0, dp_size derived as 2

Defensive patterns

Strategy: validation

Validate before calling

def validate_cp(world_size: int, cp_size: int) -> None:
    if world_size % cp_size != 0:
        raise SystemExit(f"cp_size={cp_size} must divide world_size={world_size}")

Prevention

Restrict cp_size to powers of two on typical GPU counts
Add a config linter that checks cp_size <= world_size and divisibility
Log world_size in job startup logs to debug mismatches quickly

When it happens

Trigger: Setting `cp_size: 3` (or any non-divisor) in the v1 training args while launching with e.g. 4 or 8 ranks and leaving `dp_size` unset.

Common situations: Enabling context/sequence parallelism for long-context training with a `cp_size` copied from a different GPU topology; using `cp_size` larger than world size; odd `cp_size` on power-of-two GPU counts.

Related errors

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/318c390ee3b3f06d. Report an issue: GitHub.