hiyouga/LlamaFactory · error · ValueError

`num_layers` {num_layers} should be divisible by `num_layer_

Error message

`num_layers` {num_layers} should be divisible by `num_layer_trainable` {finetuning_args.freeze_trainable_layers}.

What it means

Raised in _setup_freeze_tuning when use_llama_pro is enabled and the model's total layer count is not evenly divisible by freeze_trainable_layers. Llama-Pro expansion inserts one trainable block every stride = num_layers / freeze_trainable_layers layers, so the division must be exact.

Source

Thrown at src/llamafactory/model/adapter.py:82

        return

    logger.info_rank0("Fine-tuning method: Freeze")
    if hasattr(model.config, "text_config"):  # composite models
        config = getattr(model.config, "text_config")
    else:
        config = model.config

    num_layers = (
        getattr(config, "num_hidden_layers", None)
        or getattr(config, "num_layers", None)
        or getattr(config, "n_layer", None)
    )
    if not num_layers:
        raise ValueError("Current model does not support freeze tuning.")

    if finetuning_args.use_llama_pro:
        if num_layers % finetuning_args.freeze_trainable_layers != 0:
            raise ValueError(
                f"`num_layers` {num_layers} should be "
                f"divisible by `num_layer_trainable` {finetuning_args.freeze_trainable_layers}."
            )

        stride = num_layers // finetuning_args.freeze_trainable_layers
        trainable_layer_ids = range(stride - 1, num_layers + stride - 1, stride)
    elif finetuning_args.freeze_trainable_layers > 0:  # fine-tuning the last n layers if num_layer_trainable > 0
        trainable_layer_ids = range(max(0, num_layers - finetuning_args.freeze_trainable_layers), num_layers)
    else:  # fine-tuning the first n layers if num_layer_trainable < 0
        trainable_layer_ids = range(min(-finetuning_args.freeze_trainable_layers, num_layers))

    hidden_modules = set()
    non_hidden_modules = set()
    for name, _ in model.named_parameters():
        if ".0." in name:
            hidden_modules.add(name.split(".0.")[-1].split(".")[0])
        elif ".1." in name:  # MoD starts from layer 1
            hidden_modules.add(name.split(".1.")[-1].split(".")[0])

View on GitHub (pinned to f28afaf635)

Solutions

1. Set freeze_trainable_layers to a divisor of your model's layer count (32-layer model: 8, 16, 32...; check num_hidden_layers in config.json).
2. Check the reference: the original Llama-Pro setup expands an 8B model with freeze_trainable_layers=8 over 32 layers.
3. Disable use_llama_pro if you do not need block-expanded training.

Example fix

# before (32-layer model)
freeze_trainable_layers: 3
use_llama_pro: true

# after
freeze_trainable_layers: 8
use_llama_pro: true

from transformers import AutoConfig
cfg = AutoConfig.from_pretrained(model_path)
if hasattr(cfg, "text_config"):
    cfg = cfg.text_config
num_layers = cfg.num_hidden_layers
if use_llama_pro:
    assert num_layers % freeze_trainable_layers == 0, \
        f"{num_layers} layers not divisible by freeze_trainable_layers={freeze_trainable_layers}"

When it happens

Trigger: Config with finetuning_type: freeze, use_llama_pro: true and a freeze_trainable_layers value that does not divide num_layers (e.g. 32 layers with freeze_trainable_layers: 3).

Common situations: Reusing a Llama-Pro config written for an 8/7B model (e.g. divisible value) on a model with a different depth; hand-tuning freeze_trainable_layers without checking the layer count.

Related errors

• Current model does not support freeze tuning.
• `use_llama_pro` is only valid for Freeze or LoRA training.
• Module {} is not found, please choose from {}
• Current model does not support freeze tuning.
• Module {module_name} not found in hidden modules: {hidden_mo

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/b6a9c23926bf0748. Report an issue: GitHub.