hiyouga/LlamaFactory · critical · RuntimeError

FSDPTurbo EP mesh is not initialized.

Error message

FSDPTurbo EP mesh is not initialized.

What it means

RuntimeError raised in prepare_model_ep() when self.parallel_state.ep_mesh is None after parallel-state initialization but before expert_parallelize_modules() is called. FSDPTurbo's parallel state is expected to build an EP device mesh during initialize(); a None mesh means the EP dimension was never constructed, so expert sharding cannot proceed.

Source

Thrown at src/llamafactory/v1/plugins/trainer_plugins/distributed/fsdpturbo.py:388

        if self.ep_size > 1:
            ep_plan = EPPlanConfig(
                apply_modules=ep_modules,
                dispatcher=self.dist_config.get("ep_dispatcher", "eager"),
                apply_efsdp_modules=self._get_ep_fsdp_modules(spec),
            )
            ep_plan.gradient_divide_factor = float(self.ep_size * self.parallel_state.efsdp_size)
            fsdp_plan = FSDPPlanConfig(
                # FSDPTurbo uses this plan only to place EFSDP hooks and select its
                # implementation. EFSDP targets come from ep_plan.apply_efsdp_modules.
                apply_modules={},
                hook_modules=self.dist_config.get("hook_modules", []),
                fsdp_implementation=self.dist_config.get("fsdp_implementation", "native"),
            )
            ep_mesh = self.parallel_state.ep_mesh
            efsdp_mesh = self.parallel_state.efsdp_mesh
            if ep_mesh is None:
                raise RuntimeError("FSDPTurbo EP mesh is not initialized.")
            if self.ep_fsdp_size > 1 and efsdp_mesh is None:
                raise RuntimeError("FSDPTurbo EFSDP mesh is not initialized.")
            if self.rank == 0:
                logger.info("Applying FSDPTurbo EP backend.")
                logger.info(f"FSDPTurbo EP apply patterns: {ep_modules}")
                logger.info(f"FSDPTurbo EP device mesh: {ep_mesh}")
                logger.info(f"FSDPTurbo EP gradient divide factor: {ep_plan.gradient_divide_factor}")

            model = expert_parallelize_modules(model, ep_mesh, ep_plan)

            if self.ep_fsdp_size > 1:
                if self.rank == 0:
                    logger.info(f"FSDPTurbo EFSDP apply patterns: {ep_plan.apply_efsdp_modules}")
                    logger.info(f"FSDPTurbo EFSDP device mesh: {efsdp_mesh}")
                model = expert_fully_shard_modules(model, efsdp_mesh, ep_plan, fsdp_plan)

        # Collect ignored params for the outer FSDP wrap
        fsdp_ignored_modules = list(self.dist_config.get("fsdp_ignored_modules", []))

View on GitHub (pinned to f28afaf635)

Solutions

1. Ensure world_size is divisible by ep_size (and by ep_fsdp_size) so the EP mesh can be constructed.
2. Confirm you launch through the FSDPTurboFSDP2Engine path (name: fsdpturbo) so parallel_state.initialize() runs before prepare_model_ep().
3. Check logs from get_fsdpturbo_parallel_state()/initialize() for earlier warnings that explain why the EP mesh was not built.
4. Restart in a fresh process; a previously initialized parallel state with different sizes will not rebuild meshes.

engine = FSDPTurboFSDP2Engine(dist_config)
assert engine.parallel_state.ep_mesh is not None, "EP mesh missing; check world_size % ep_size == 0"

When it happens

Trigger: ep_size > 1 in the dist_config while the FSDPTurbo parallel state failed or was skipped during engine construction (fsdpturbo.py:280-282 calls parallel_state.initialize only in FSDPTurboFSDP2Engine.__init__). Also when a stale/already-initialized parallel state from a previous run is reused with a different ep_size.

Common situations: Mismatch between ep_size and world size (world_size not divisible by ep_size) causing initialize() to skip mesh creation; instantiating the engine in a process where torch.distributed was not yet initialized; mixing FSDPTurbo with another distributed backend that owns the device mesh.

Related errors

• FSDPTurbo EFSDP mesh is not initialized.
• No FSDPTurbo EP spec is registered for model_type={_get_mode
• `tensor_model_parallel_size` must be >= 1.
• `pipeline_model_parallel_size` must be >= 1.
• Disable FSDP activation checkpointing when using KTransforme

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/485e22e9ed52bba7. Report an issue: GitHub.