{"record":{"id":"485e22e9ed52bba7","repo":"hiyouga/LlamaFactory","slug":"fsdpturbo-ep-mesh-is-not-initialized","errorCode":null,"errorMessage":"FSDPTurbo EP mesh is not initialized.","messagePattern":"FSDPTurbo EP mesh is not initialized\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"src/llamafactory/v1/plugins/trainer_plugins/distributed/fsdpturbo.py","lineNumber":388,"sourceCode":"\n        if self.ep_size > 1:\n            ep_plan = EPPlanConfig(\n                apply_modules=ep_modules,\n                dispatcher=self.dist_config.get(\"ep_dispatcher\", \"eager\"),\n                apply_efsdp_modules=self._get_ep_fsdp_modules(spec),\n            )\n            ep_plan.gradient_divide_factor = float(self.ep_size * self.parallel_state.efsdp_size)\n            fsdp_plan = FSDPPlanConfig(\n                # FSDPTurbo uses this plan only to place EFSDP hooks and select its\n                # implementation. EFSDP targets come from ep_plan.apply_efsdp_modules.\n                apply_modules={},\n                hook_modules=self.dist_config.get(\"hook_modules\", []),\n                fsdp_implementation=self.dist_config.get(\"fsdp_implementation\", \"native\"),\n            )\n            ep_mesh = self.parallel_state.ep_mesh\n            efsdp_mesh = self.parallel_state.efsdp_mesh\n            if ep_mesh is None:\n                raise RuntimeError(\"FSDPTurbo EP mesh is not initialized.\")\n            if self.ep_fsdp_size > 1 and efsdp_mesh is None:\n                raise RuntimeError(\"FSDPTurbo EFSDP mesh is not initialized.\")\n            if self.rank == 0:\n                logger.info(\"Applying FSDPTurbo EP backend.\")\n                logger.info(f\"FSDPTurbo EP apply patterns: {ep_modules}\")\n                logger.info(f\"FSDPTurbo EP device mesh: {ep_mesh}\")\n                logger.info(f\"FSDPTurbo EP gradient divide factor: {ep_plan.gradient_divide_factor}\")\n\n            model = expert_parallelize_modules(model, ep_mesh, ep_plan)\n\n            if self.ep_fsdp_size > 1:\n                if self.rank == 0:\n                    logger.info(f\"FSDPTurbo EFSDP apply patterns: {ep_plan.apply_efsdp_modules}\")\n                    logger.info(f\"FSDPTurbo EFSDP device mesh: {efsdp_mesh}\")\n                model = expert_fully_shard_modules(model, efsdp_mesh, ep_plan, fsdp_plan)\n\n        # Collect ignored params for the outer FSDP wrap\n        fsdp_ignored_modules = list(self.dist_config.get(\"fsdp_ignored_modules\", []))","sourceCodeStart":370,"sourceCodeEnd":406,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/plugins/trainer_plugins/distributed/fsdpturbo.py#L370-L406","documentation":"RuntimeError raised in prepare_model_ep() when self.parallel_state.ep_mesh is None after parallel-state initialization but before expert_parallelize_modules() is called. FSDPTurbo's parallel state is expected to build an EP device mesh during initialize(); a None mesh means the EP dimension was never constructed, so expert sharding cannot proceed.","triggerScenarios":"ep_size > 1 in the dist_config while the FSDPTurbo parallel state failed or was skipped during engine construction (fsdpturbo.py:280-282 calls parallel_state.initialize only in FSDPTurboFSDP2Engine.__init__). Also when a stale/already-initialized parallel state from a previous run is reused with a different ep_size.","commonSituations":"Mismatch between ep_size and world size (world_size not divisible by ep_size) causing initialize() to skip mesh creation; instantiating the engine in a process where torch.distributed was not yet initialized; mixing FSDPTurbo with another distributed backend that owns the device mesh.","solutions":["Ensure world_size is divisible by ep_size (and by ep_fsdp_size) so the EP mesh can be constructed.","Confirm you launch through the FSDPTurboFSDP2Engine path (name: fsdpturbo) so parallel_state.initialize() runs before prepare_model_ep().","Check logs from get_fsdpturbo_parallel_state()/initialize() for earlier warnings that explain why the EP mesh was not built.","Restart in a fresh process; a previously initialized parallel state with different sizes will not rebuild meshes."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"engine = FSDPTurboFSDP2Engine(dist_config)\nassert engine.parallel_state.ep_mesh is not None, \"EP mesh missing; check world_size % ep_size == 0\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always launch FSDPTurbo through its engine so parallel_state.initialize() runs.","Validate world_size divisibility by ep_size * ep_fsdp_size in the launch script.","Never reuse a process for two runs with different parallel sizes."],"tags":["distributed","fsdp","expert-parallelism","device-mesh"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}