hiyouga/LlamaFactory · error · RuntimeError

FSDPTurbo expert parallelism requires an initialized distrib

Error message

FSDPTurbo expert parallelism requires an initialized distributed device mesh.

What it means

When ep_size > 1, FSDPTurbo slices the existing DP device mesh to build the expert mesh (edp x efsdp x ep x expert_cp). If the distributed interface reports no initialized DP device mesh, there is nothing to derive expert process groups from, and initialization fails. The DP mesh must be created (e.g. by the accelerator/distributed bootstrap) before enabling expert parallelism.

Source

Thrown at src/llamafactory/v1/plugins/trainer_plugins/distributed/fsdpturbo.py:81

        topology = (dp_size, cp_size, ep_size)
        if self._initialized:
            current_topology = (self.dp_size, self.cp_size, self.ep_size)
            if topology != current_topology:
                raise RuntimeError(
                    f"FSDPTurbo parallel state is already initialized with {current_topology}, got {topology}."
                )
            return

        self.dp_size = dp_size
        self.cp_size = cp_size
        self.ep_size = ep_size

        if ep_size > 1:
            self.efsdp_size = dp_size // ep_size
            self.edp_size = dp_size // (ep_size * self.efsdp_size)
            if dist_interface.get_device_mesh(Dim.DP) is None:
                raise RuntimeError("FSDPTurbo expert parallelism requires an initialized distributed device mesh.")

            self.expert_mesh = init_device_mesh(
                device_type=dist_interface.current_device.type,
                mesh_shape=(self.edp_size, self.efsdp_size, self.ep_size, self.cp_size),
                mesh_dim_names=(self.EDP, self.EFSDP, self.EP, self.EXPERT_CP),
            )
            self.edp_mesh = self.expert_mesh[self.EDP]
            self.efsdp_mesh = self.expert_mesh[self.EFSDP]
            self.ep_mesh = self.expert_mesh[self.EP]
            self.expert_cp_mesh = self.expert_mesh[self.EXPERT_CP]

        self._initialized = True


_FSDPTURBO_PARALLEL_STATE = FSDPTurboParallelState()


def get_fsdpturbo_parallel_state() -> FSDPTurboParallelState:

View on GitHub (pinned to f28afaf635)

Solutions

1. Use the standard trainer/accelerator entry point so the DP mesh is initialized before plugin setup
2. In custom code, create the DP DeviceMesh (init_device_mesh) before calling initialize() with ep_size > 1
3. If expert parallelism is optional for you, set ep_size: 1

Example fix

# before
state.initialize(iface, {"ep_size": 4})  # DP mesh never created

# after
import torch.distributed as dist
from torch.distributed.device_mesh import init_device_mesh
dist.init_process_group(backend="nccl")
dp_mesh = init_device_mesh("cuda", (dist.get_world_size(),), mesh_dim_names=("dp",))
state.initialize(iface, {"ep_size": 4})

ep = int(dist_config.get("ep_size", 1))
if ep > 1:
    assert dist_interface.get_device_mesh(Dim.DP) is not None, \
        "initialize the DP device mesh (accelerator bootstrap) before enabling ep_size > 1"

When it happens

Trigger: Initializing FSDPTurbo with ep_size > 1 before the base DP DeviceMesh was created — dist_interface.get_device_mesh(Dim.DP) returns None.

Common situations: Custom integration that constructs the FSDPTurbo state directly instead of going through the standard accelerator bootstrap; bootstrap ordering changes; mocking the distributed interface in tests without providing a mesh.

Related errors

• ep_size must be positive, got {ep_size}.
• dp_size must be divisible by ep_size, got {dp_size} % {ep_si
• FSDPTurbo parallel state must be initialized before clipping
• world_size ({helper.get_world_size()}) must be divisible by
• mp_replicate_size * mp_shard_size must equal to world_size,

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/7231ee2e5a15313e. Report an issue: GitHub.