hiyouga/LlamaFactory · error · ValueError

Model was not supported.

Error message

Model was not supported.

What it means

find_expanded_modules (misc.py) maps num_layer_trainable onto the model by reading config.num_hidden_layers. If the config lacks num_hidden_layers (getattr returns None), LlamaFactory cannot compute per-block LoRA target layers for the expanded-blocks training mode and raises the generic ValueError 'Model was not supported.'.

Source

Thrown at src/llamafactory/model/model_utils/misc.py:59

        forbidden_modules.update(COMPOSITE_MODELS[model_type].vision_model_keys)

    module_names = set()
    for name, module in model.named_modules():
        if any(forbidden_module in name for forbidden_module in forbidden_modules):
            continue

        if "Linear" in module.__class__.__name__ and "Embedding" not in module.__class__.__name__:
            module_names.add(name.split(".")[-1])

    logger.info_rank0("Found linear modules: {}".format(",".join(module_names)))
    return list(module_names)


def find_expanded_modules(model: "PreTrainedModel", target_modules: list[str], num_layer_trainable: int) -> list[str]:
    r"""Find the modules in the expanded blocks to apply lora."""
    num_layers = getattr(model.config, "num_hidden_layers", None)
    if not num_layers:
        raise ValueError("Model was not supported.")

    if num_layers % num_layer_trainable != 0:
        raise ValueError(
            f"`num_layers` {num_layers} should be divisible by `num_layer_trainable` {num_layer_trainable}."
        )

    stride = num_layers // num_layer_trainable
    trainable_layer_ids = range(stride - 1, num_layers + stride - 1, stride)
    trainable_layers = [f".{idx:d}." for idx in trainable_layer_ids]
    module_names = []
    for name, _ in model.named_modules():
        if any(target_module in name for target_module in target_modules) and any(
            trainable_layer in name for trainable_layer in trainable_layers
        ):
            module_names.append(name)

    logger.info_rank0("Apply lora to layers: {}.".format(",".join(map(str, trainable_layer_ids))))
    return module_names

View on GitHub (pinned to f28afaf635)

Solutions

1. Do not use num_layer_trainable for this model; rely on standard lora_target over all layers instead.
2. If it is your model, expose num_hidden_layers on the top-level config (copy from text_config) before loading.
3. Check config.to_dict() for where layer count lives and file/patch support for that model_type.
4. Switch to a supported model family when you need the expanded-blocks training mode.

Example fix

# before
finetuning_args.num_layer_trainable = 4  # config lacks num_hidden_layers -> ValueError

# after (custom model fix)
config.num_hidden_layers = config.text_config.num_hidden_layers
# or drop num_layer_trainable and use plain lora_target

num_layers = getattr(model.config, "num_hidden_layers", None)
if finetuning_args.num_layer_trainable > 0:
    assert num_layers, "config.num_hidden_layers missing; num_layer_trainable unsupported for this model"

When it happens

Trigger: Setting finetuning_args.num_layer_trainable (train only every N-th block's LoRA) on a model whose config does not define num_hidden_layers — e.g. some multimodal models that store depth under vision_config/text_config or num_layers, or custom configs with different key names.

Common situations: Using num_layer_trainable with a new/custom architecture; models where the layer count lives in a nested config; experimental models added without a patcher entry exposing num_hidden_layers.

Related errors

• `reward_model_type` cannot be lora for Freeze/Full PPO train
• Cannot use LoRA with GaLore, APOLLO or BAdam together.
• Cannot use PiSSA for current training stage.
• `loraplus_lr_ratio` is only valid for LoRA training.
• `use_rslora` is only valid for LoRA training.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/794dda28bacb9ec7. Report an issue: GitHub.