{"record":{"id":"e75423aa09fb63a7","repo":"hiyouga/LlamaFactory","slug":"current-model-is-not-supported-by-mixture-of-depth","errorCode":null,"errorMessage":"Current model is not supported by mixture-of-depth.","messagePattern":"Current model is not supported by mixture-of-depth\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/model/model_utils/mod.py","lineNumber":38,"sourceCode":"if TYPE_CHECKING:\n    from transformers import PretrainedConfig, PreTrainedModel\n\n    from ...hparams import ModelArguments\n\n\ndef load_mod_pretrained_model(**init_kwargs) -> \"PreTrainedModel\":\n    from MoD import AutoMoDModelForCausalLM\n\n    return AutoMoDModelForCausalLM.from_pretrained(**init_kwargs)\n\n\ndef convert_pretrained_model_to_mod(\n    model: \"PreTrainedModel\", config: \"PretrainedConfig\", model_args: \"ModelArguments\"\n) -> \"PreTrainedModel\":\n    from MoD import apply_mod_to_hf\n\n    if getattr(config, \"model_type\", None) not in MOD_SUPPORTED_MODELS:\n        raise ValueError(\"Current model is not supported by mixture-of-depth.\")\n\n    model = apply_mod_to_hf(model)\n    model = model.to(model_args.compute_dtype)\n    return model\n","sourceCodeStart":20,"sourceCodeEnd":43,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/model/model_utils/mod.py#L20-L43","documentation":"convert_pretrained_model_to_mod (mod.py) applies the Mixture-of-Depths transformation via MoD.apply_mod_to_hf, but only for architectures listed in MOD_SUPPORTED_MODELS. Any other config.model_type raises ValueError before conversion. MoD rewrites decoder layers with a capacity-based router and is architecture-sensitive, hence the whitelist.","triggerScenarios":"Enabling mixture-of-depths (finetuning_args/model_args with mod settings, e.g. mod_depth) on a model whose config.model_type is not in MOD_SUPPORTED_MODELS (check constants for the list — typically llama and a few others).","commonSituations":"Trying MoD experiments on newer architectures (qwen, gemma, mistral variants) that were never adapted; running example MoD configs against an unsupported base checkpoint.","solutions":["Use one of the supported base architectures listed in MOD_SUPPORTED_MODELS (see src/llamafactory/model/model_utils/mod.py or constants).","Port/extend MoD support for your architecture — verify layer structure compatibility first, then add the model_type to MOD_SUPPORTED_MODELS and test.","Disable the mixture-of-depths options if you do not need MoD."],"exampleFix":"# before\nmodel_name_or_path: Qwen/Qwen2.5-7B   # not in MOD_SUPPORTED_MODELS\nmod_depth: 8\n\n# after\nmodel_name_or_path: meta-llama/Llama-3-8B  # supported by MoD\nmod_depth: 8","handlingStrategy":"type-guard","validationCode":"from llamafactory.model.model_utils.mod import MOD_SUPPORTED_MODELS\n\nassert getattr(config, \"model_type\", None) in MOD_SUPPORTED_MODELS, (\n    f\"MoD supports {MOD_SUPPORTED_MODELS}; got {getattr(config, 'model_type', None)}\"\n)","typeGuard":"def is_mod_supported(config) -> bool:\n    from llamafactory.model.model_utils.mod import MOD_SUPPORTED_MODELS\n    return getattr(config, \"model_type\", None) in MOD_SUPPORTED_MODELS","tryCatchPattern":null,"preventionTips":["Check MOD_SUPPORTED_MODELS before planning MoD experiments.","Treat MoD configs as architecture-specific; never reuse across model families untested."],"tags":["mixture-of-depths","architecture","model-support"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}