hiyouga/LlamaFactory · error · ValueError

Current model is not supported by mixture-of-depth.

Error message

Current model is not supported by mixture-of-depth.

What it means

convert_pretrained_model_to_mod (mod.py) applies the Mixture-of-Depths transformation via MoD.apply_mod_to_hf, but only for architectures listed in MOD_SUPPORTED_MODELS. Any other config.model_type raises ValueError before conversion. MoD rewrites decoder layers with a capacity-based router and is architecture-sensitive, hence the whitelist.

Source

Thrown at src/llamafactory/model/model_utils/mod.py:38

if TYPE_CHECKING:
    from transformers import PretrainedConfig, PreTrainedModel

    from ...hparams import ModelArguments


def load_mod_pretrained_model(**init_kwargs) -> "PreTrainedModel":
    from MoD import AutoMoDModelForCausalLM

    return AutoMoDModelForCausalLM.from_pretrained(**init_kwargs)


def convert_pretrained_model_to_mod(
    model: "PreTrainedModel", config: "PretrainedConfig", model_args: "ModelArguments"
) -> "PreTrainedModel":
    from MoD import apply_mod_to_hf

    if getattr(config, "model_type", None) not in MOD_SUPPORTED_MODELS:
        raise ValueError("Current model is not supported by mixture-of-depth.")

    model = apply_mod_to_hf(model)
    model = model.to(model_args.compute_dtype)
    return model

View on GitHub (pinned to f28afaf635)

Solutions

1. Use one of the supported base architectures listed in MOD_SUPPORTED_MODELS (see src/llamafactory/model/model_utils/mod.py or constants).
2. Port/extend MoD support for your architecture — verify layer structure compatibility first, then add the model_type to MOD_SUPPORTED_MODELS and test.
3. Disable the mixture-of-depths options if you do not need MoD.

Example fix

# before
model_name_or_path: Qwen/Qwen2.5-7B   # not in MOD_SUPPORTED_MODELS
mod_depth: 8

# after
model_name_or_path: meta-llama/Llama-3-8B  # supported by MoD
mod_depth: 8

from llamafactory.model.model_utils.mod import MOD_SUPPORTED_MODELS

assert getattr(config, "model_type", None) in MOD_SUPPORTED_MODELS, (
    f"MoD supports {MOD_SUPPORTED_MODELS}; got {getattr(config, 'model_type', None)}"
)

def is_mod_supported(config) -> bool:
    from llamafactory.model.model_utils.mod import MOD_SUPPORTED_MODELS
    return getattr(config, "model_type", None) in MOD_SUPPORTED_MODELS

When it happens

Trigger: Enabling mixture-of-depths (finetuning_args/model_args with mod settings, e.g. mod_depth) on a model whose config.model_type is not in MOD_SUPPORTED_MODELS (check constants for the list — typically llama and a few others).

Common situations: Trying MoD experiments on newer architectures (qwen, gemma, mistral variants) that were never adapted; running example MoD configs against an unsupported base checkpoint.

Related errors

• Invalid API key.
• Invalid length
• Current model does not support freeze tuning.
• Module {} is not found, please choose from {}
• ChatGLM model is not supported yet.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/e75423aa09fb63a7. Report an issue: GitHub.