{"record":{"id":"38a2ab8b937ce6c1","repo":"hiyouga/LlamaFactory","slug":"no-fsdpturbo-ep-spec-is-registered-for-model-type","errorCode":null,"errorMessage":"No FSDPTurbo EP spec is registered for model_type={_get_model_type(model)}.","messagePattern":"No FSDPTurbo EP spec is registered for model_type=(.+?)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/v1/plugins/trainer_plugins/distributed/fsdpturbo.py","lineNumber":366,"sourceCode":"                dim = shard_placement.dim\n                slices[dim] = slice(0, sliced_tensor.shape[dim])\n            local_tensor[tuple(slices)].copy_(sliced_tensor)\n            return\n\n        param.data.copy_(loaded_tensor)\n\n    def prepare_model_ep(self, model: HFModel) -> tuple[HFModel, set]:\n        \"\"\"Apply FSDPTurbo EP/EFSDP and return parameters excluded from outer FSDP.\"\"\"\n        from fsdp_turbo.distributed.expert_parallel.expert_fully_shard_parallel import (\n            expert_fully_shard_modules,\n        )\n        from fsdp_turbo.distributed.expert_parallel.expert_parallel import expert_parallelize_modules\n        from fsdp_turbo.fsdp_turbo_config import EPPlanConfig, FSDPPlanConfig\n        from fsdp_turbo.utils.str_match import module_name_match\n\n        spec = FSDPTurboEPModelSpec.get(model)\n        if spec is None:\n            raise ValueError(f\"No FSDPTurbo EP spec is registered for model_type={_get_model_type(model)}.\")\n\n        ep_modules = spec.ep_modules\n        model = spec.prepare(model)\n\n        if self.ep_size > 1:\n            ep_plan = EPPlanConfig(\n                apply_modules=ep_modules,\n                dispatcher=self.dist_config.get(\"ep_dispatcher\", \"eager\"),\n                apply_efsdp_modules=self._get_ep_fsdp_modules(spec),\n            )\n            ep_plan.gradient_divide_factor = float(self.ep_size * self.parallel_state.efsdp_size)\n            fsdp_plan = FSDPPlanConfig(\n                # FSDPTurbo uses this plan only to place EFSDP hooks and select its\n                # implementation. EFSDP targets come from ep_plan.apply_efsdp_modules.\n                apply_modules={},\n                hook_modules=self.dist_config.get(\"hook_modules\", []),\n                fsdp_implementation=self.dist_config.get(\"fsdp_implementation\", \"native\"),\n            )","sourceCodeStart":348,"sourceCodeEnd":384,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/plugins/trainer_plugins/distributed/fsdpturbo.py#L348-L384","documentation":"Raised by FSDPTurboDistributed.prepare_model_ep() when FSDPTurboEPModelSpec.get(model) returns None, meaning the model's config.model_type has no entry in the EP spec registry. The registry (fsdpturbo.py:194-266) only knows 'qwen3_moe' and 'qwen3_5_moe'; expert parallelism cannot be applied to any other architecture. The error is a hard stop before any sharding occurs.","triggerScenarios":"Setting the fsdpturbo distributed backend with ep_size > 1 while training a model whose model_type is not 'qwen3_moe' or 'qwen3_5_moe' (e.g. deepseek_v3, llama4_moe, mixtral). Also triggered when the model config lacks a model_type attribute, since FSDPTurboEPModelSpec.get() returns None for model_type=None.","commonSituations":"User points an FSDPTurbo EP config at a newly released MoE model that LlamaFactory has not yet added a spec for; or a custom/merged model whose config.json has a nonstandard model_type string.","solutions":["Set ep_size: 1 (or remove the fsdpturbo EP options) so prepare_model_ep's EP path is skipped and the model trains with plain FSDP.","Switch to a supported model family (qwen3_moe or qwen3_5_moe) if expert parallelism is required.","Register a spec for your model_type via the FSDPTurboEPModelSpec.register(model_type, ep_modules=[...], ep_fsdp_modules=[...]) decorator in fsdpturbo.py, listing the expert-module name patterns, then rebuild.","If the model_type exists but the check fails, verify model.config.model_type in a debugger to confirm what string the registry lookup used."],"exampleFix":"# before (config.yaml)\ndist_config:\n  name: fsdpturbo\n  ep_size: 8\n\n# after (config.yaml) - model not in registry, disable EP\ndist_config:\n  name: fsdpturbo\n  ep_size: 1","handlingStrategy":"validation","validationCode":"from llamafactory.v1.plugins.trainer_plugins.distributed.fsdpturbo import FSDPTurboEPModelSpec, _get_model_type\nmt = _get_model_type(model)\nif ep_size > 1 and (mt is None or mt not in FSDPTurboEPModelSpec._registry):\n    raise SystemExit(f\"model_type={mt!r} has no FSDPTurbo EP spec; set ep_size=1 or register a spec\")","typeGuard":"def supports_fsdpturbo_ep(model) -> bool:\n    mt = _get_model_type(model)\n    return mt is not None and mt in FSDPTurboEPModelSpec._registry","tryCatchPattern":null,"preventionTips":["Check FSDPTurboEPModelSpec._registry keys before configuring ep_size > 1.","Keep configs per model family instead of sharing one EP config across architectures."],"tags":["distributed","fsdp","expert-parallelism","moe","config"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}