hiyouga/LlamaFactory · error · ValueError

`num_layers` {num_layers} should be divisible by `num_layer_

Error message

`num_layers` {num_layers} should be divisible by `num_layer_trainable` {num_layer_trainable}.

What it means

In the expanded-blocks LoRA mode (num_layer_trainable), LlamaFactory distributes trainable layers evenly across the network: it requires the model's layer count to be an exact multiple of num_layer_trainable so each trainable block covers the same stride. If num_layers % num_layer_trainable != 0 it raises ValueError showing both values.

Source

Thrown at src/llamafactory/model/model_utils/misc.py:62

    for name, module in model.named_modules():
        if any(forbidden_module in name for forbidden_module in forbidden_modules):
            continue

        if "Linear" in module.__class__.__name__ and "Embedding" not in module.__class__.__name__:
            module_names.add(name.split(".")[-1])

    logger.info_rank0("Found linear modules: {}".format(",".join(module_names)))
    return list(module_names)


def find_expanded_modules(model: "PreTrainedModel", target_modules: list[str], num_layer_trainable: int) -> list[str]:
    r"""Find the modules in the expanded blocks to apply lora."""
    num_layers = getattr(model.config, "num_hidden_layers", None)
    if not num_layers:
        raise ValueError("Model was not supported.")

    if num_layers % num_layer_trainable != 0:
        raise ValueError(
            f"`num_layers` {num_layers} should be divisible by `num_layer_trainable` {num_layer_trainable}."
        )

    stride = num_layers // num_layer_trainable
    trainable_layer_ids = range(stride - 1, num_layers + stride - 1, stride)
    trainable_layers = [f".{idx:d}." for idx in trainable_layer_ids]
    module_names = []
    for name, _ in model.named_modules():
        if any(target_module in name for target_module in target_modules) and any(
            trainable_layer in name for trainable_layer in trainable_layers
        ):
            module_names.append(name)

    logger.info_rank0("Apply lora to layers: {}.".format(",".join(map(str, trainable_layer_ids))))
    return module_names


def register_autoclass(config: "PretrainedConfig", model: "PreTrainedModel", tokenizer: "PreTrainedTokenizer"):

View on GitHub (pinned to f28afaf635)

Solutions

1. Pick a divisor of the layer count: for 32 layers use 2/4/8/16; for 28 use 2/4/7/14; check config.num_hidden_layers first.
2. Compute it dynamically: num_layer_trainable = largest divisor of num_hidden_layers that meets your budget.
3. Drop num_layer_trainable and use full lora_target if divisibility cannot be met.

Example fix

# before (mistral-7b, 32 layers is fine, but for a 28-layer model)
num_layer_trainable: 5   # 28 % 5 != 0 -> ValueError

# after
num_layer_trainable: 4   # 28 % 4 == 0

n = getattr(model.config, "num_hidden_layers", 0)
assert n % finetuning_args.num_layer_trainable == 0, (
    f"num_layer_trainable must divide {n}; got {finetuning_args.num_layer_trainable}"
)

When it happens

Trigger: finetuning_args.num_layer_trainable set to a value that does not divide config.num_hidden_layers evenly — e.g. a 32-layer model with num_layer_trainable=5, or a 28-layer model (num_layer_trainable=4).

Common situations: Copying num_layer_trainable: 4 from an example config tuned for 32-layer LLaMA onto a 28-layer Mistral or 24-layer model; power-of-two habits colliding with non-power-of-two depths.

Related errors

• Model was not supported.
• `reward_model_type` cannot be lora for Freeze/Full PPO train
• Cannot use LoRA with GaLore, APOLLO or BAdam together.
• Cannot use PiSSA for current training stage.
• `loraplus_lr_ratio` is only valid for LoRA training.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/9dd3c89ec9f8172b. Report an issue: GitHub.