{"record":{"id":"a28efda47523dc1c","repo":"hiyouga/LlamaFactory","slug":"mp-replicate-size-mp-shard-size-must-equal-to-wo","errorCode":null,"errorMessage":"mp_replicate_size * mp_shard_size must equal to world_size, got {self.mp_replicate_size} * {self.mp_shard_size} != {helper.get_world_size()}.","messagePattern":"mp_replicate_size \\* mp_shard_size must equal to world_size, got (.+?) \\* (.+?) != (.+?)\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/v1/accelerator/interface.py","lineNumber":82,"sourceCode":"    mp_shard_size: int | None = None\n    \"\"\"Model parallel shard size, default to world_size // mp_replicate_size.\"\"\"\n    dp_size: int | None = None\n    \"\"\"Data parallel size, default to world_size // cp_size.\"\"\"\n    cp_size: int = 1\n    \"\"\"Context parallel size, default to 1.\"\"\"\n\n    def __post_init__(self) -> None:\n        if not helper.is_distributed():\n            self.mp_shard_size = 1\n        elif self.mp_shard_size is None:\n            if helper.get_world_size() % self.mp_replicate_size != 0:\n                raise ValueError(\n                    f\"world_size ({helper.get_world_size()}) must be divisible by \"\n                    f\"mp_replicate_size ({self.mp_replicate_size}).\"\n                )\n            self.mp_shard_size = helper.get_world_size() // self.mp_replicate_size\n        elif self.mp_replicate_size * self.mp_shard_size != helper.get_world_size():\n            raise ValueError(\n                f\"mp_replicate_size * mp_shard_size must equal to world_size, \"\n                f\"got {self.mp_replicate_size} * {self.mp_shard_size} != {helper.get_world_size()}.\"\n            )\n\n        if not helper.is_distributed():\n            self.dp_size = 1\n        elif self.dp_size is None:\n            if helper.get_world_size() % self.cp_size != 0:\n                raise ValueError(\n                    f\"world_size ({helper.get_world_size()}) must be divisible by cp_size ({self.cp_size}).\"\n                )\n            self.dp_size = helper.get_world_size() // self.cp_size\n        elif self.dp_size * self.cp_size != helper.get_world_size():\n            raise ValueError(\n                f\"dp_size * cp_size must equal to world_size, \"\n                f\"got {self.dp_size} * {self.cp_size} != {helper.get_world_size()}.\"\n            )\n","sourceCodeStart":64,"sourceCodeEnd":100,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/accelerator/interface.py#L64-L100","documentation":"Raised in `DistributedConfig.__post_init__` when BOTH `mp_replicate_size` and `mp_shard_size` are explicitly provided in a distributed run but their product does not equal world size. The mesh must exactly cover all ranks, so `mp_replicate_size * mp_shard_size == world_size` is an invariant.","triggerScenarios":"Explicitly configuring both `mp_replicate_size` and `mp_shard_size` in the v1 dist config while launching with a different number of processes, e.g. 2x4 mesh on 8 ranks launched with `--nproc_per_node=4` on one node.","commonSituations":"Reusing a multi-node config on fewer nodes; resizing GPU count without updating the mesh; assuming the trainer will silently clamp extra ranks.","solutions":["Adjust one of the two sizes so the product equals world size (e.g. for 4 ranks use 2x2 or 1x4)","Or drop `mp_shard_size` from the config and let it be derived as `world_size // mp_replicate_size` (which triggers the friendlier divisibility check instead)","Verify world size first: `echo $WORLD_SIZE` or check the launcher's process count"],"exampleFix":"# before: single node, 4 GPUs\ndist:\n  mp_replicate_size: 2\n  mp_shard_size: 4   # 2*4=8 != 4\n\n# after\ndist:\n  mp_replicate_size: 2\n  mp_shard_size: 2   # 2*2=4 == world_size","handlingStrategy":"validation","validationCode":"def validate_mesh(world_size: int, mp_replicate: int, mp_shard: int) -> None:\n    if mp_replicate * mp_shard != world_size:\n        raise SystemExit(\n            f\"mp mesh {mp_replicate}x{mp_shard}={mp_replicate * mp_shard} != world_size {world_size}\"\n        )","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Set only mp_replicate_size and let mp_shard_size be derived","Validate the product in the job template before submitting to a scheduler","Regenerate configs on GPU-count changes instead of editing by hand"],"tags":["distributed-training","v1","configuration","device-mesh"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}