hiyouga/LlamaFactory · error · ValueError

dp_size * cp_size must equal to world_size, got {self.dp_siz

Error message

dp_size * cp_size must equal to world_size, got {self.dp_size} * {self.cp_size} != {helper.get_world_size()}.

What it means

Raised in `DistributedConfig.__post_init__` when both `dp_size` and `cp_size` are set explicitly in a distributed run but `dp_size * cp_size != world_size`. The data mesh must consume exactly the ranks not covered by the model mesh, hence the strict equality.

Source

Thrown at src/llamafactory/v1/accelerator/interface.py:96

                    f"mp_replicate_size ({self.mp_replicate_size})."
                )
            self.mp_shard_size = helper.get_world_size() // self.mp_replicate_size
        elif self.mp_replicate_size * self.mp_shard_size != helper.get_world_size():
            raise ValueError(
                f"mp_replicate_size * mp_shard_size must equal to world_size, "
                f"got {self.mp_replicate_size} * {self.mp_shard_size} != {helper.get_world_size()}."
            )

        if not helper.is_distributed():
            self.dp_size = 1
        elif self.dp_size is None:
            if helper.get_world_size() % self.cp_size != 0:
                raise ValueError(
                    f"world_size ({helper.get_world_size()}) must be divisible by cp_size ({self.cp_size})."
                )
            self.dp_size = helper.get_world_size() // self.cp_size
        elif self.dp_size * self.cp_size != helper.get_world_size():
            raise ValueError(
                f"dp_size * cp_size must equal to world_size, "
                f"got {self.dp_size} * {self.cp_size} != {helper.get_world_size()}."
            )

    @property
    def model_mesh_shape(self) -> tuple[int, int]:
        """Model parallel mesh shape."""
        return (self.mp_replicate_size, self.mp_shard_size)

    @property
    def model_mesh_dim_names(self) -> tuple[str, str]:
        """Model parallel mesh dimension names."""
        return (Dim.MP_REPLICATE.value, Dim.MP_SHARD.value)

    @property
    def data_mesh_shape(self) -> tuple[int, int]:
        """Data parallel mesh shape."""
        return (self.dp_size, self.cp_size)

View on GitHub (pinned to f28afaf635)

Solutions

Recompute and fix the pair so `dp_size * cp_size == world_size`
Or remove `dp_size` from config and let it be derived (`world_size // cp_size`)
Confirm actual world size from the launcher logs (`WORLD_SIZE` env) before choosing values

Example fix

# before: launched with torchrun --nproc_per_node=4
dist:
  dp_size: 4
  cp_size: 2   # 4*2=8 != 4

# after
dist:
  dp_size: 2
  cp_size: 2   # 2*2=4 == world_size

Defensive patterns

Strategy: validation

Validate before calling

def validate_dp_cp(world_size: int, dp_size: int, cp_size: int) -> None:
    if dp_size * cp_size != world_size:
        raise SystemExit(f"dp*cp = {dp_size}*{cp_size} != world_size {world_size}")

Prevention

Omit dp_size and let it be derived from cp_size
Compute all parallel dims in one place (a `compute_parallel_layout(world_size)` helper) so they stay consistent
Unit-test config generation for each cluster topology you run

When it happens

Trigger: Explicitly configuring `dp_size` and `cp_size` (e.g. dp=4, cp=2) while launching with a world size of 8 via a launcher that only starts 4 processes, or vice versa.

Common situations: Scaling a job up/down (changing `--nproc_per_node` or node count) without updating dp/cp; leftover explicit `dp_size` from an earlier experiment now conflicting with a new `cp_size`.

Related errors

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/34eeb4e9c2703408. Report an issue: GitHub.