{"record":{"id":"1d3f6b7105e6e1e3","repo":"hiyouga/LlamaFactory","slug":"fsdpturbo-efsdp-mesh-is-not-initialized","errorCode":null,"errorMessage":"FSDPTurbo EFSDP mesh is not initialized.","messagePattern":"FSDPTurbo EFSDP mesh is not initialized\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"src/llamafactory/v1/plugins/trainer_plugins/distributed/fsdpturbo.py","lineNumber":390,"sourceCode":"            ep_plan = EPPlanConfig(\n                apply_modules=ep_modules,\n                dispatcher=self.dist_config.get(\"ep_dispatcher\", \"eager\"),\n                apply_efsdp_modules=self._get_ep_fsdp_modules(spec),\n            )\n            ep_plan.gradient_divide_factor = float(self.ep_size * self.parallel_state.efsdp_size)\n            fsdp_plan = FSDPPlanConfig(\n                # FSDPTurbo uses this plan only to place EFSDP hooks and select its\n                # implementation. EFSDP targets come from ep_plan.apply_efsdp_modules.\n                apply_modules={},\n                hook_modules=self.dist_config.get(\"hook_modules\", []),\n                fsdp_implementation=self.dist_config.get(\"fsdp_implementation\", \"native\"),\n            )\n            ep_mesh = self.parallel_state.ep_mesh\n            efsdp_mesh = self.parallel_state.efsdp_mesh\n            if ep_mesh is None:\n                raise RuntimeError(\"FSDPTurbo EP mesh is not initialized.\")\n            if self.ep_fsdp_size > 1 and efsdp_mesh is None:\n                raise RuntimeError(\"FSDPTurbo EFSDP mesh is not initialized.\")\n            if self.rank == 0:\n                logger.info(\"Applying FSDPTurbo EP backend.\")\n                logger.info(f\"FSDPTurbo EP apply patterns: {ep_modules}\")\n                logger.info(f\"FSDPTurbo EP device mesh: {ep_mesh}\")\n                logger.info(f\"FSDPTurbo EP gradient divide factor: {ep_plan.gradient_divide_factor}\")\n\n            model = expert_parallelize_modules(model, ep_mesh, ep_plan)\n\n            if self.ep_fsdp_size > 1:\n                if self.rank == 0:\n                    logger.info(f\"FSDPTurbo EFSDP apply patterns: {ep_plan.apply_efsdp_modules}\")\n                    logger.info(f\"FSDPTurbo EFSDP device mesh: {efsdp_mesh}\")\n                model = expert_fully_shard_modules(model, efsdp_mesh, ep_plan, fsdp_plan)\n\n        # Collect ignored params for the outer FSDP wrap\n        fsdp_ignored_modules = list(self.dist_config.get(\"fsdp_ignored_modules\", []))\n        if self.ep_size > 1:\n            fsdp_ignored_modules.extend(ep_modules)","sourceCodeStart":372,"sourceCodeEnd":408,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/plugins/trainer_plugins/distributed/fsdpturbo.py#L372-L408","documentation":"RuntimeError raised when ep_fsdp_size > 1 (EFSDP enabled) but parallel_state.efsdp_mesh is None. EFSDP shards expert weights on a dedicated sub-mesh orthogonal to EP; without the mesh, expert_fully_shard_modules cannot be applied. The EP mesh check at line 388 passing while this fails isolates the problem to the EFSDP dimension specifically.","triggerScenarios":"Configuring ep_fsdp_size > 1 in the fsdpturbo dist_config while the parallel state only built the EP mesh (e.g. world_size not divisible by ep_size * ep_fsdp_size), so no EFSDP sub-mesh exists.","commonSituations":"User scales ep_size up without rescaling ep_fsdp_size, breaking the divisibility of world size; or runs on fewer GPUs than the config assumes.","solutions":["Make world_size divisible by ep_size * ep_fsdp_size (e.g. 8 GPUs with ep_size 4 and ep_fsdp_size 2).","If EFSDP is not needed, set ep_fsdp_size: 1 so the EFSDP branch and this check are skipped.","Verify the logged device-mesh layout at startup matches the requested EP/EFSDP split before training begins."],"exampleFix":"# before (8 GPUs)\ndist_config:\n  name: fsdpturbo\n  ep_size: 4\n  ep_fsdp_size: 4   # 4*4=16 > 8, mesh cannot be built\n\n# after\ndist_config:\n  name: fsdpturbo\n  ep_size: 4\n  ep_fsdp_size: 2","handlingStrategy":"validation","validationCode":"world = torch.distributed.get_world_size()\nassert world % (ep_size * ep_fsdp_size) == 0, f\"world_size {world} not divisible by ep_size*ep_fsdp_size {ep_size*ep_fsdp_size}\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Assert world_size % (ep_size * ep_fsdp_size) == 0 before launching.","Set ep_fsdp_size: 1 unless EFSDP is explicitly needed."],"tags":["distributed","fsdp","expert-parallelism","device-mesh","config"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}