{"record":{"id":"7231ee2e5a15313e","repo":"hiyouga/LlamaFactory","slug":"fsdpturbo-expert-parallelism-requires-an-initializ","errorCode":null,"errorMessage":"FSDPTurbo expert parallelism requires an initialized distributed device mesh.","messagePattern":"FSDPTurbo expert parallelism requires an initialized distributed device mesh\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/v1/plugins/trainer_plugins/distributed/fsdpturbo.py","lineNumber":81,"sourceCode":"\n        topology = (dp_size, cp_size, ep_size)\n        if self._initialized:\n            current_topology = (self.dp_size, self.cp_size, self.ep_size)\n            if topology != current_topology:\n                raise RuntimeError(\n                    f\"FSDPTurbo parallel state is already initialized with {current_topology}, got {topology}.\"\n                )\n            return\n\n        self.dp_size = dp_size\n        self.cp_size = cp_size\n        self.ep_size = ep_size\n\n        if ep_size > 1:\n            self.efsdp_size = dp_size // ep_size\n            self.edp_size = dp_size // (ep_size * self.efsdp_size)\n            if dist_interface.get_device_mesh(Dim.DP) is None:\n                raise RuntimeError(\"FSDPTurbo expert parallelism requires an initialized distributed device mesh.\")\n\n            self.expert_mesh = init_device_mesh(\n                device_type=dist_interface.current_device.type,\n                mesh_shape=(self.edp_size, self.efsdp_size, self.ep_size, self.cp_size),\n                mesh_dim_names=(self.EDP, self.EFSDP, self.EP, self.EXPERT_CP),\n            )\n            self.edp_mesh = self.expert_mesh[self.EDP]\n            self.efsdp_mesh = self.expert_mesh[self.EFSDP]\n            self.ep_mesh = self.expert_mesh[self.EP]\n            self.expert_cp_mesh = self.expert_mesh[self.EXPERT_CP]\n\n        self._initialized = True\n\n\n_FSDPTURBO_PARALLEL_STATE = FSDPTurboParallelState()\n\n\ndef get_fsdpturbo_parallel_state() -> FSDPTurboParallelState:","sourceCodeStart":63,"sourceCodeEnd":99,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/plugins/trainer_plugins/distributed/fsdpturbo.py#L63-L99","documentation":"When ep_size > 1, FSDPTurbo slices the existing DP device mesh to build the expert mesh (edp x efsdp x ep x expert_cp). If the distributed interface reports no initialized DP device mesh, there is nothing to derive expert process groups from, and initialization fails. The DP mesh must be created (e.g. by the accelerator/distributed bootstrap) before enabling expert parallelism.","triggerScenarios":"Initializing FSDPTurbo with ep_size > 1 before the base DP DeviceMesh was created — dist_interface.get_device_mesh(Dim.DP) returns None.","commonSituations":"Custom integration that constructs the FSDPTurbo state directly instead of going through the standard accelerator bootstrap; bootstrap ordering changes; mocking the distributed interface in tests without providing a mesh.","solutions":["Use the standard trainer/accelerator entry point so the DP mesh is initialized before plugin setup","In custom code, create the DP DeviceMesh (init_device_mesh) before calling initialize() with ep_size > 1","If expert parallelism is optional for you, set ep_size: 1"],"exampleFix":"# before\nstate.initialize(iface, {\"ep_size\": 4})  # DP mesh never created\n\n# after\nimport torch.distributed as dist\nfrom torch.distributed.device_mesh import init_device_mesh\ndist.init_process_group(backend=\"nccl\")\ndp_mesh = init_device_mesh(\"cuda\", (dist.get_world_size(),), mesh_dim_names=(\"dp\",))\nstate.initialize(iface, {\"ep_size\": 4})","handlingStrategy":"validation","validationCode":"ep = int(dist_config.get(\"ep_size\", 1))\nif ep > 1:\n    assert dist_interface.get_device_mesh(Dim.DP) is not None, \\\n        \"initialize the DP device mesh (accelerator bootstrap) before enabling ep_size > 1\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always go through the standard trainer bootstrap order before plugin init","In tests, build a minimal DeviceMesh before exercising EP code paths"],"tags":["fsdpturbo","expert-parallel","device-mesh","initialization-order"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}