{"record":{"id":"b6a9c23926bf0748","repo":"hiyouga/LlamaFactory","slug":"num-layers-num-layers-should-be-divisible-by-b6a9c2","errorCode":null,"errorMessage":"`num_layers` {num_layers} should be divisible by `num_layer_trainable` {finetuning_args.freeze_trainable_layers}.","messagePattern":"`num_layers` (.+?) should be divisible by `num_layer_trainable` (.+?)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/model/adapter.py","lineNumber":82,"sourceCode":"        return\n\n    logger.info_rank0(\"Fine-tuning method: Freeze\")\n    if hasattr(model.config, \"text_config\"):  # composite models\n        config = getattr(model.config, \"text_config\")\n    else:\n        config = model.config\n\n    num_layers = (\n        getattr(config, \"num_hidden_layers\", None)\n        or getattr(config, \"num_layers\", None)\n        or getattr(config, \"n_layer\", None)\n    )\n    if not num_layers:\n        raise ValueError(\"Current model does not support freeze tuning.\")\n\n    if finetuning_args.use_llama_pro:\n        if num_layers % finetuning_args.freeze_trainable_layers != 0:\n            raise ValueError(\n                f\"`num_layers` {num_layers} should be \"\n                f\"divisible by `num_layer_trainable` {finetuning_args.freeze_trainable_layers}.\"\n            )\n\n        stride = num_layers // finetuning_args.freeze_trainable_layers\n        trainable_layer_ids = range(stride - 1, num_layers + stride - 1, stride)\n    elif finetuning_args.freeze_trainable_layers > 0:  # fine-tuning the last n layers if num_layer_trainable > 0\n        trainable_layer_ids = range(max(0, num_layers - finetuning_args.freeze_trainable_layers), num_layers)\n    else:  # fine-tuning the first n layers if num_layer_trainable < 0\n        trainable_layer_ids = range(min(-finetuning_args.freeze_trainable_layers, num_layers))\n\n    hidden_modules = set()\n    non_hidden_modules = set()\n    for name, _ in model.named_parameters():\n        if \".0.\" in name:\n            hidden_modules.add(name.split(\".0.\")[-1].split(\".\")[0])\n        elif \".1.\" in name:  # MoD starts from layer 1\n            hidden_modules.add(name.split(\".1.\")[-1].split(\".\")[0])","sourceCodeStart":64,"sourceCodeEnd":100,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/model/adapter.py#L64-L100","documentation":"Raised in _setup_freeze_tuning when use_llama_pro is enabled and the model's total layer count is not evenly divisible by freeze_trainable_layers. Llama-Pro expansion inserts one trainable block every stride = num_layers / freeze_trainable_layers layers, so the division must be exact.","triggerScenarios":"Config with finetuning_type: freeze, use_llama_pro: true and a freeze_trainable_layers value that does not divide num_layers (e.g. 32 layers with freeze_trainable_layers: 3).","commonSituations":"Reusing a Llama-Pro config written for an 8/7B model (e.g. divisible value) on a model with a different depth; hand-tuning freeze_trainable_layers without checking the layer count.","solutions":["Set freeze_trainable_layers to a divisor of your model's layer count (32-layer model: 8, 16, 32...; check num_hidden_layers in config.json).","Check the reference: the original Llama-Pro setup expands an 8B model with freeze_trainable_layers=8 over 32 layers.","Disable use_llama_pro if you do not need block-expanded training."],"exampleFix":"# before (32-layer model)\nfreeze_trainable_layers: 3\nuse_llama_pro: true\n\n# after\nfreeze_trainable_layers: 8\nuse_llama_pro: true","handlingStrategy":"validation","validationCode":"from transformers import AutoConfig\ncfg = AutoConfig.from_pretrained(model_path)\nif hasattr(cfg, \"text_config\"):\n    cfg = cfg.text_config\nnum_layers = cfg.num_hidden_layers\nif use_llama_pro:\n    assert num_layers % freeze_trainable_layers == 0, \\\n        f\"{num_layers} layers not divisible by freeze_trainable_layers={freeze_trainable_layers}\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Read num_hidden_layers from config.json before setting use_llama_pro values.","Keep the reference pair (32 layers / 8 blocks) in mind when porting Llama-Pro configs."],"tags":["llama-pro","freeze-tuning","layer-count","divisibility"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}