hiyouga/LlamaFactory · error · ValueError

No FSDPTurbo EP spec is registered for model_type={_get_mode

Error message

No FSDPTurbo EP spec is registered for model_type={_get_model_type(model)}.

What it means

Raised by FSDPTurboDistributed.prepare_model_ep() when FSDPTurboEPModelSpec.get(model) returns None, meaning the model's config.model_type has no entry in the EP spec registry. The registry (fsdpturbo.py:194-266) only knows 'qwen3_moe' and 'qwen3_5_moe'; expert parallelism cannot be applied to any other architecture. The error is a hard stop before any sharding occurs.

Source

Thrown at src/llamafactory/v1/plugins/trainer_plugins/distributed/fsdpturbo.py:366

                dim = shard_placement.dim
                slices[dim] = slice(0, sliced_tensor.shape[dim])
            local_tensor[tuple(slices)].copy_(sliced_tensor)
            return

        param.data.copy_(loaded_tensor)

    def prepare_model_ep(self, model: HFModel) -> tuple[HFModel, set]:
        """Apply FSDPTurbo EP/EFSDP and return parameters excluded from outer FSDP."""
        from fsdp_turbo.distributed.expert_parallel.expert_fully_shard_parallel import (
            expert_fully_shard_modules,
        )
        from fsdp_turbo.distributed.expert_parallel.expert_parallel import expert_parallelize_modules
        from fsdp_turbo.fsdp_turbo_config import EPPlanConfig, FSDPPlanConfig
        from fsdp_turbo.utils.str_match import module_name_match

        spec = FSDPTurboEPModelSpec.get(model)
        if spec is None:
            raise ValueError(f"No FSDPTurbo EP spec is registered for model_type={_get_model_type(model)}.")

        ep_modules = spec.ep_modules
        model = spec.prepare(model)

        if self.ep_size > 1:
            ep_plan = EPPlanConfig(
                apply_modules=ep_modules,
                dispatcher=self.dist_config.get("ep_dispatcher", "eager"),
                apply_efsdp_modules=self._get_ep_fsdp_modules(spec),
            )
            ep_plan.gradient_divide_factor = float(self.ep_size * self.parallel_state.efsdp_size)
            fsdp_plan = FSDPPlanConfig(
                # FSDPTurbo uses this plan only to place EFSDP hooks and select its
                # implementation. EFSDP targets come from ep_plan.apply_efsdp_modules.
                apply_modules={},
                hook_modules=self.dist_config.get("hook_modules", []),
                fsdp_implementation=self.dist_config.get("fsdp_implementation", "native"),
            )

View on GitHub (pinned to f28afaf635)

Solutions

1. Set ep_size: 1 (or remove the fsdpturbo EP options) so prepare_model_ep's EP path is skipped and the model trains with plain FSDP.
2. Switch to a supported model family (qwen3_moe or qwen3_5_moe) if expert parallelism is required.
3. Register a spec for your model_type via the FSDPTurboEPModelSpec.register(model_type, ep_modules=[...], ep_fsdp_modules=[...]) decorator in fsdpturbo.py, listing the expert-module name patterns, then rebuild.
4. If the model_type exists but the check fails, verify model.config.model_type in a debugger to confirm what string the registry lookup used.

Example fix

# before (config.yaml)
dist_config:
  name: fsdpturbo
  ep_size: 8

# after (config.yaml) - model not in registry, disable EP
dist_config:
  name: fsdpturbo
  ep_size: 1

from llamafactory.v1.plugins.trainer_plugins.distributed.fsdpturbo import FSDPTurboEPModelSpec, _get_model_type
mt = _get_model_type(model)
if ep_size > 1 and (mt is None or mt not in FSDPTurboEPModelSpec._registry):
    raise SystemExit(f"model_type={mt!r} has no FSDPTurbo EP spec; set ep_size=1 or register a spec")

def supports_fsdpturbo_ep(model) -> bool:
    mt = _get_model_type(model)
    return mt is not None and mt in FSDPTurboEPModelSpec._registry

When it happens

Trigger: Setting the fsdpturbo distributed backend with ep_size > 1 while training a model whose model_type is not 'qwen3_moe' or 'qwen3_5_moe' (e.g. deepseek_v3, llama4_moe, mixtral). Also triggered when the model config lacks a model_type attribute, since FSDPTurboEPModelSpec.get() returns None for model_type=None.

Common situations: User points an FSDPTurbo EP config at a newly released MoE model that LlamaFactory has not yet added a spec for; or a custom/merged model whose config.json has a nonstandard model_type string.

Related errors

• FSDPTurbo EFSDP mesh is not initialized.
• FSDPTurbo EP mesh is not initialized.
• `tensor_model_parallel_size` must be >= 1.
• `pipeline_model_parallel_size` must be >= 1.
• `expert_model_parallel_size` must be >= 1.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/38a2ab8b937ce6c1. Report an issue: GitHub.