{"record":{"id":"34eeb4e9c2703408","repo":"hiyouga/LlamaFactory","slug":"dp-size-cp-size-must-equal-to-world-size-got-s","errorCode":null,"errorMessage":"dp_size * cp_size must equal to world_size, got {self.dp_size} * {self.cp_size} != {helper.get_world_size()}.","messagePattern":"dp_size \\* cp_size must equal to world_size, got (.+?) \\* (.+?) != (.+?)\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/v1/accelerator/interface.py","lineNumber":96,"sourceCode":"                    f\"mp_replicate_size ({self.mp_replicate_size}).\"\n                )\n            self.mp_shard_size = helper.get_world_size() // self.mp_replicate_size\n        elif self.mp_replicate_size * self.mp_shard_size != helper.get_world_size():\n            raise ValueError(\n                f\"mp_replicate_size * mp_shard_size must equal to world_size, \"\n                f\"got {self.mp_replicate_size} * {self.mp_shard_size} != {helper.get_world_size()}.\"\n            )\n\n        if not helper.is_distributed():\n            self.dp_size = 1\n        elif self.dp_size is None:\n            if helper.get_world_size() % self.cp_size != 0:\n                raise ValueError(\n                    f\"world_size ({helper.get_world_size()}) must be divisible by cp_size ({self.cp_size}).\"\n                )\n            self.dp_size = helper.get_world_size() // self.cp_size\n        elif self.dp_size * self.cp_size != helper.get_world_size():\n            raise ValueError(\n                f\"dp_size * cp_size must equal to world_size, \"\n                f\"got {self.dp_size} * {self.cp_size} != {helper.get_world_size()}.\"\n            )\n\n    @property\n    def model_mesh_shape(self) -> tuple[int, int]:\n        \"\"\"Model parallel mesh shape.\"\"\"\n        return (self.mp_replicate_size, self.mp_shard_size)\n\n    @property\n    def model_mesh_dim_names(self) -> tuple[str, str]:\n        \"\"\"Model parallel mesh dimension names.\"\"\"\n        return (Dim.MP_REPLICATE.value, Dim.MP_SHARD.value)\n\n    @property\n    def data_mesh_shape(self) -> tuple[int, int]:\n        \"\"\"Data parallel mesh shape.\"\"\"\n        return (self.dp_size, self.cp_size)","sourceCodeStart":78,"sourceCodeEnd":114,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/accelerator/interface.py#L78-L114","documentation":"Raised in `DistributedConfig.__post_init__` when both `dp_size` and `cp_size` are set explicitly in a distributed run but `dp_size * cp_size != world_size`. The data mesh must consume exactly the ranks not covered by the model mesh, hence the strict equality.","triggerScenarios":"Explicitly configuring `dp_size` and `cp_size` (e.g. dp=4, cp=2) while launching with a world size of 8 via a launcher that only starts 4 processes, or vice versa.","commonSituations":"Scaling a job up/down (changing `--nproc_per_node` or node count) without updating dp/cp; leftover explicit `dp_size` from an earlier experiment now conflicting with a new `cp_size`.","solutions":["Recompute and fix the pair so `dp_size * cp_size == world_size`","Or remove `dp_size` from config and let it be derived (`world_size // cp_size`)","Confirm actual world size from the launcher logs (`WORLD_SIZE` env) before choosing values"],"exampleFix":"# before: launched with torchrun --nproc_per_node=4\ndist:\n  dp_size: 4\n  cp_size: 2   # 4*2=8 != 4\n\n# after\ndist:\n  dp_size: 2\n  cp_size: 2   # 2*2=4 == world_size","handlingStrategy":"validation","validationCode":"def validate_dp_cp(world_size: int, dp_size: int, cp_size: int) -> None:\n    if dp_size * cp_size != world_size:\n        raise SystemExit(f\"dp*cp = {dp_size}*{cp_size} != world_size {world_size}\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Omit dp_size and let it be derived from cp_size","Compute all parallel dims in one place (a `compute_parallel_layout(world_size)` helper) so they stay consistent","Unit-test config generation for each cluster topology you run"],"tags":["distributed-training","v1","configuration","context-parallelism"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}