hiyouga/LlamaFactory · critical · RuntimeError

FSDPTurbo EFSDP mesh is not initialized.

Error message

FSDPTurbo EFSDP mesh is not initialized.

What it means

RuntimeError raised when ep_fsdp_size > 1 (EFSDP enabled) but parallel_state.efsdp_mesh is None. EFSDP shards expert weights on a dedicated sub-mesh orthogonal to EP; without the mesh, expert_fully_shard_modules cannot be applied. The EP mesh check at line 388 passing while this fails isolates the problem to the EFSDP dimension specifically.

Source

Thrown at src/llamafactory/v1/plugins/trainer_plugins/distributed/fsdpturbo.py:390

            ep_plan = EPPlanConfig(
                apply_modules=ep_modules,
                dispatcher=self.dist_config.get("ep_dispatcher", "eager"),
                apply_efsdp_modules=self._get_ep_fsdp_modules(spec),
            )
            ep_plan.gradient_divide_factor = float(self.ep_size * self.parallel_state.efsdp_size)
            fsdp_plan = FSDPPlanConfig(
                # FSDPTurbo uses this plan only to place EFSDP hooks and select its
                # implementation. EFSDP targets come from ep_plan.apply_efsdp_modules.
                apply_modules={},
                hook_modules=self.dist_config.get("hook_modules", []),
                fsdp_implementation=self.dist_config.get("fsdp_implementation", "native"),
            )
            ep_mesh = self.parallel_state.ep_mesh
            efsdp_mesh = self.parallel_state.efsdp_mesh
            if ep_mesh is None:
                raise RuntimeError("FSDPTurbo EP mesh is not initialized.")
            if self.ep_fsdp_size > 1 and efsdp_mesh is None:
                raise RuntimeError("FSDPTurbo EFSDP mesh is not initialized.")
            if self.rank == 0:
                logger.info("Applying FSDPTurbo EP backend.")
                logger.info(f"FSDPTurbo EP apply patterns: {ep_modules}")
                logger.info(f"FSDPTurbo EP device mesh: {ep_mesh}")
                logger.info(f"FSDPTurbo EP gradient divide factor: {ep_plan.gradient_divide_factor}")

            model = expert_parallelize_modules(model, ep_mesh, ep_plan)

            if self.ep_fsdp_size > 1:
                if self.rank == 0:
                    logger.info(f"FSDPTurbo EFSDP apply patterns: {ep_plan.apply_efsdp_modules}")
                    logger.info(f"FSDPTurbo EFSDP device mesh: {efsdp_mesh}")
                model = expert_fully_shard_modules(model, efsdp_mesh, ep_plan, fsdp_plan)

        # Collect ignored params for the outer FSDP wrap
        fsdp_ignored_modules = list(self.dist_config.get("fsdp_ignored_modules", []))
        if self.ep_size > 1:
            fsdp_ignored_modules.extend(ep_modules)

View on GitHub (pinned to f28afaf635)

Solutions

1. Make world_size divisible by ep_size * ep_fsdp_size (e.g. 8 GPUs with ep_size 4 and ep_fsdp_size 2).
2. If EFSDP is not needed, set ep_fsdp_size: 1 so the EFSDP branch and this check are skipped.
3. Verify the logged device-mesh layout at startup matches the requested EP/EFSDP split before training begins.

Example fix

# before (8 GPUs)
dist_config:
  name: fsdpturbo
  ep_size: 4
  ep_fsdp_size: 4   # 4*4=16 > 8, mesh cannot be built

# after
dist_config:
  name: fsdpturbo
  ep_size: 4
  ep_fsdp_size: 2

world = torch.distributed.get_world_size()
assert world % (ep_size * ep_fsdp_size) == 0, f"world_size {world} not divisible by ep_size*ep_fsdp_size {ep_size*ep_fsdp_size}"

When it happens

Trigger: Configuring ep_fsdp_size > 1 in the fsdpturbo dist_config while the parallel state only built the EP mesh (e.g. world_size not divisible by ep_size * ep_fsdp_size), so no EFSDP sub-mesh exists.

Common situations: User scales ep_size up without rescaling ep_fsdp_size, breaking the divisibility of world size; or runs on fewer GPUs than the config assumes.

Related errors

• No FSDPTurbo EP spec is registered for model_type={_get_mode
• FSDPTurbo EP mesh is not initialized.
• `tensor_model_parallel_size` must be >= 1.
• `pipeline_model_parallel_size` must be >= 1.
• Disable FSDP activation checkpointing when using KTransforme

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/1d3f6b7105e6e1e3. Report an issue: GitHub.