{"record":{"id":"9dd3c89ec9f8172b","repo":"hiyouga/LlamaFactory","slug":"num-layers-num-layers-should-be-divisible-by-9dd3c8","errorCode":null,"errorMessage":"`num_layers` {num_layers} should be divisible by `num_layer_trainable` {num_layer_trainable}.","messagePattern":"`num_layers` (.+?) should be divisible by `num_layer_trainable` (.+?)\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/model/model_utils/misc.py","lineNumber":62,"sourceCode":"    for name, module in model.named_modules():\n        if any(forbidden_module in name for forbidden_module in forbidden_modules):\n            continue\n\n        if \"Linear\" in module.__class__.__name__ and \"Embedding\" not in module.__class__.__name__:\n            module_names.add(name.split(\".\")[-1])\n\n    logger.info_rank0(\"Found linear modules: {}\".format(\",\".join(module_names)))\n    return list(module_names)\n\n\ndef find_expanded_modules(model: \"PreTrainedModel\", target_modules: list[str], num_layer_trainable: int) -> list[str]:\n    r\"\"\"Find the modules in the expanded blocks to apply lora.\"\"\"\n    num_layers = getattr(model.config, \"num_hidden_layers\", None)\n    if not num_layers:\n        raise ValueError(\"Model was not supported.\")\n\n    if num_layers % num_layer_trainable != 0:\n        raise ValueError(\n            f\"`num_layers` {num_layers} should be divisible by `num_layer_trainable` {num_layer_trainable}.\"\n        )\n\n    stride = num_layers // num_layer_trainable\n    trainable_layer_ids = range(stride - 1, num_layers + stride - 1, stride)\n    trainable_layers = [f\".{idx:d}.\" for idx in trainable_layer_ids]\n    module_names = []\n    for name, _ in model.named_modules():\n        if any(target_module in name for target_module in target_modules) and any(\n            trainable_layer in name for trainable_layer in trainable_layers\n        ):\n            module_names.append(name)\n\n    logger.info_rank0(\"Apply lora to layers: {}.\".format(\",\".join(map(str, trainable_layer_ids))))\n    return module_names\n\n\ndef register_autoclass(config: \"PretrainedConfig\", model: \"PreTrainedModel\", tokenizer: \"PreTrainedTokenizer\"):","sourceCodeStart":44,"sourceCodeEnd":80,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/model/model_utils/misc.py#L44-L80","documentation":"In the expanded-blocks LoRA mode (num_layer_trainable), LlamaFactory distributes trainable layers evenly across the network: it requires the model's layer count to be an exact multiple of num_layer_trainable so each trainable block covers the same stride. If num_layers % num_layer_trainable != 0 it raises ValueError showing both values.","triggerScenarios":"finetuning_args.num_layer_trainable set to a value that does not divide config.num_hidden_layers evenly — e.g. a 32-layer model with num_layer_trainable=5, or a 28-layer model (num_layer_trainable=4).","commonSituations":"Copying num_layer_trainable: 4 from an example config tuned for 32-layer LLaMA onto a 28-layer Mistral or 24-layer model; power-of-two habits colliding with non-power-of-two depths.","solutions":["Pick a divisor of the layer count: for 32 layers use 2/4/8/16; for 28 use 2/4/7/14; check config.num_hidden_layers first.","Compute it dynamically: num_layer_trainable = largest divisor of num_hidden_layers that meets your budget.","Drop num_layer_trainable and use full lora_target if divisibility cannot be met."],"exampleFix":"# before (mistral-7b, 32 layers is fine, but for a 28-layer model)\nnum_layer_trainable: 5   # 28 % 5 != 0 -> ValueError\n\n# after\nnum_layer_trainable: 4   # 28 % 4 == 0","handlingStrategy":"validation","validationCode":"n = getattr(model.config, \"num_hidden_layers\", 0)\nassert n % finetuning_args.num_layer_trainable == 0, (\n    f\"num_layer_trainable must divide {n}; got {finetuning_args.num_layer_trainable}\"\n)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Derive num_layer_trainable from config.num_hidden_layers (pick a divisor) rather than hard-coding.","Log num_hidden_layers at config time so mismatches are obvious before launch."],"tags":["lora","num-layer-trainable","config-validation"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}