{"record":{"id":"3f7eb96ecd7fed70","repo":"hiyouga/LlamaFactory","slug":"cannot-merge-adapters-to-a-quantized-model","errorCode":null,"errorMessage":"Cannot merge adapters to a quantized model.","messagePattern":"Cannot merge adapters to a quantized model\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/train/tuner.py","lineNumber":192,"sourceCode":"\n\ndef export_model(args: Optional[dict[str, Any]] = None) -> None:\n    model_args, data_args, finetuning_args, _ = get_infer_args(args)\n\n    if model_args.export_dir is None:\n        raise ValueError(\"Please specify `export_dir` to save model.\")\n\n    if model_args.adapter_name_or_path is not None and model_args.export_quantization_bit is not None:\n        raise ValueError(\"Please merge adapters before quantizing the model.\")\n\n    tokenizer_module = load_tokenizer(model_args)\n    tokenizer = tokenizer_module[\"tokenizer\"]\n    processor = tokenizer_module[\"processor\"]\n    template = get_template_and_fix_tokenizer(tokenizer, data_args)\n    model = load_model(tokenizer, model_args, finetuning_args)  # must after fixing tokenizer to resize vocab\n\n    if getattr(model, \"quantization_method\", None) is not None and model_args.adapter_name_or_path is not None:\n        raise ValueError(\"Cannot merge adapters to a quantized model.\")\n\n    if not isinstance(model, PreTrainedModel):\n        raise ValueError(\"The model is not a `PreTrainedModel`, export aborted.\")\n\n    if getattr(model, \"quantization_method\", None) is not None:  # quantized model adopts float16 type\n        setattr(model.config, \"torch_dtype\", torch.float16)\n    else:\n        if model_args.infer_dtype == \"auto\":\n            output_dtype = getattr(model.config, \"torch_dtype\", torch.float32)\n            if output_dtype == torch.float32:  # if infer_dtype is auto, try using half precision first\n                output_dtype = infer_optim_dtype(torch.bfloat16)\n        else:\n            output_dtype = getattr(torch, model_args.infer_dtype)\n\n        setattr(model.config, \"torch_dtype\", output_dtype)\n        model = model.to(output_dtype)\n        logger.info_rank0(f\"Convert model dtype to: {output_dtype}.\")\n","sourceCodeStart":174,"sourceCodeEnd":210,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/train/tuner.py#L174-L210","documentation":"export_model (src/llamafactory/train/tuner.py:192) checks the loaded model for a `quantization_method` attribute; if the base model itself was loaded quantized (e.g. `quantization_bit` in the config or a pre-quantized checkpoint) AND an adapter is requested, merging is refused — LoRA weights cannot be folded into low-bit (bnb/GPTQ/AWQ) weights.","triggerScenarios":"Export config where the model is loaded quantized (quantization_bit set, or model_name_or_path points at a 4/8-bit or GPTQ checkpoint) while adapter_name_or_path is also set.","commonSituations":"Fine-tuning on top of a pre-quantized community checkpoint (GPTQ/AWQ) and then trying to export a merged model.","solutions":["Export the adapter only: omit adapter merging by exporting the adapter directory as-is (or leave adapter_name_or_path set but drop the quantized base, using the full-precision base).","Load the full-precision base model (no quantization_bit, fp/bf16 checkpoint) for the merge export.","Keep the quantized base + adapter separate and apply the adapter at inference time (PEFT loading), which is the supported pattern for quantized bases."],"exampleFix":"# before (yaml)\nmodel_name_or_path: TheBloke/model-GPTQ   # quantized base\nadapter_name_or_path: output/lora\nexport_dir: out\n\n# after: merge against full-precision base\nmodel_name_or_path: meta-llama/Llama-3-8b   # fp/bf16\nadapter_name_or_path: output/lora\nexport_dir: out","handlingStrategy":"validation","validationCode":"def mergeable(model, adapter_name_or_path) -> bool:\n    return adapter_name_or_path is None or getattr(model, \"quantization_method\", None) is None","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always merge adapters against the full-precision base checkpoint.","For quantized bases, keep the adapter unmerged and load it via PEFT at inference."],"tags":["export","quantization","lora","merge"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}