{"record":{"id":"744a57006415a97c","repo":"hiyouga/LlamaFactory","slug":"dp-size-must-be-divisible-by-ep-size-got-dp-size","errorCode":null,"errorMessage":"dp_size must be divisible by ep_size, got {dp_size} % {ep_size} != 0.","messagePattern":"dp_size must be divisible by ep_size, got (.+?) % (.+?) != 0\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/v1/plugins/trainer_plugins/distributed/fsdpturbo.py","lineNumber":62,"sourceCode":"        self.expert_mesh: DeviceMesh | None = None\n        self.edp_mesh: DeviceMesh | None = None\n        self.efsdp_mesh: DeviceMesh | None = None\n        self.ep_mesh: DeviceMesh | None = None\n        self.expert_cp_mesh: DeviceMesh | None = None\n\n    @property\n    def initialized(self) -> bool:\n        return self._initialized\n\n    def initialize(self, dist_interface: DistributedInterface, dist_config: dict) -> None:\n        dp_size = dist_interface.get_world_size(Dim.DP)\n        cp_size = dist_interface.strategy.cp_size\n        ep_size = int(dist_config.get(\"ep_size\", 1))\n\n        if ep_size < 1:\n            raise ValueError(f\"ep_size must be positive, got {ep_size}.\")\n        if dp_size % ep_size != 0:\n            raise ValueError(f\"dp_size must be divisible by ep_size, got {dp_size} % {ep_size} != 0.\")\n\n        topology = (dp_size, cp_size, ep_size)\n        if self._initialized:\n            current_topology = (self.dp_size, self.cp_size, self.ep_size)\n            if topology != current_topology:\n                raise RuntimeError(\n                    f\"FSDPTurbo parallel state is already initialized with {current_topology}, got {topology}.\"\n                )\n            return\n\n        self.dp_size = dp_size\n        self.cp_size = cp_size\n        self.ep_size = ep_size\n\n        if ep_size > 1:\n            self.efsdp_size = dp_size // ep_size\n            self.edp_size = dp_size // (ep_size * self.efsdp_size)\n            if dist_interface.get_device_mesh(Dim.DP) is None:","sourceCodeStart":44,"sourceCodeEnd":80,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/plugins/trainer_plugins/distributed/fsdpturbo.py#L44-L80","documentation":"FSDPTurbo builds an expert mesh of shape (edp, efsdp, ep, cp) from dp_size and ep_size; this requires dp_size to divide evenly by ep_size. If world size is not a multiple of the requested expert-parallel degree, ranks cannot be partitioned and initialization aborts.","triggerScenarios":"e.g. world_size=8 with default dp_size=8 and ep_size=3, or any combination where dp_size % ep_size != 0.","commonSituations":"User sets ep_size to the number of experts or number of nodes rather than a divisor of data-parallel size; changes GPU count (torchrun nproc) without revisiting ep_size; tensor/other parallel dims shrink dp_size unexpectedly.","solutions":["Choose ep_size that divides dp_size (from a dp of 8: 1, 2, 4, 8)","Adjust world size or other parallel dims so the division is exact","Remember dp_size here is the DP world size after other dims, not total GPUs"],"exampleFix":"# before (8 GPUs, dp=8)\ntorchrun --nproc_per_node 8 train.py  # dist_config: {\"ep_size\": 3}\n\n# after\ndist_config = {\"ep_size\": 4}  # 8 % 4 == 0","handlingStrategy":"validation","validationCode":"dp = dist_interface.get_world_size(Dim.DP)\nep = int(dist_config.get(\"ep_size\", 1))\nassert dp % ep == 0, f\"dp_size={dp} not divisible by ep_size={ep}; valid ep: {[d for d in range(1, dp + 1) if dp % d == 0]}\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Compute valid ep divisors from dp_size in launch scripts","Re-validate parallel config whenever world size changes"],"tags":["fsdpturbo","expert-parallel","distributed","topology"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}