hiyouga/LlamaFactory · error · ValueError

Cannot merge adapters to a quantized model.

Error message

Cannot merge adapters to a quantized model.

What it means

export_model (src/llamafactory/train/tuner.py:192) checks the loaded model for a `quantization_method` attribute; if the base model itself was loaded quantized (e.g. `quantization_bit` in the config or a pre-quantized checkpoint) AND an adapter is requested, merging is refused — LoRA weights cannot be folded into low-bit (bnb/GPTQ/AWQ) weights.

Source

Thrown at src/llamafactory/train/tuner.py:192

def export_model(args: Optional[dict[str, Any]] = None) -> None:
    model_args, data_args, finetuning_args, _ = get_infer_args(args)

    if model_args.export_dir is None:
        raise ValueError("Please specify `export_dir` to save model.")

    if model_args.adapter_name_or_path is not None and model_args.export_quantization_bit is not None:
        raise ValueError("Please merge adapters before quantizing the model.")

    tokenizer_module = load_tokenizer(model_args)
    tokenizer = tokenizer_module["tokenizer"]
    processor = tokenizer_module["processor"]
    template = get_template_and_fix_tokenizer(tokenizer, data_args)
    model = load_model(tokenizer, model_args, finetuning_args)  # must after fixing tokenizer to resize vocab

    if getattr(model, "quantization_method", None) is not None and model_args.adapter_name_or_path is not None:
        raise ValueError("Cannot merge adapters to a quantized model.")

    if not isinstance(model, PreTrainedModel):
        raise ValueError("The model is not a `PreTrainedModel`, export aborted.")

    if getattr(model, "quantization_method", None) is not None:  # quantized model adopts float16 type
        setattr(model.config, "torch_dtype", torch.float16)
    else:
        if model_args.infer_dtype == "auto":
            output_dtype = getattr(model.config, "torch_dtype", torch.float32)
            if output_dtype == torch.float32:  # if infer_dtype is auto, try using half precision first
                output_dtype = infer_optim_dtype(torch.bfloat16)
        else:
            output_dtype = getattr(torch, model_args.infer_dtype)

        setattr(model.config, "torch_dtype", output_dtype)
        model = model.to(output_dtype)
        logger.info_rank0(f"Convert model dtype to: {output_dtype}.")

View on GitHub (pinned to f28afaf635)

Solutions

1. Export the adapter only: omit adapter merging by exporting the adapter directory as-is (or leave adapter_name_or_path set but drop the quantized base, using the full-precision base).
2. Load the full-precision base model (no quantization_bit, fp/bf16 checkpoint) for the merge export.
3. Keep the quantized base + adapter separate and apply the adapter at inference time (PEFT loading), which is the supported pattern for quantized bases.

Example fix

# before (yaml)
model_name_or_path: TheBloke/model-GPTQ   # quantized base
adapter_name_or_path: output/lora
export_dir: out

# after: merge against full-precision base
model_name_or_path: meta-llama/Llama-3-8b   # fp/bf16
adapter_name_or_path: output/lora
export_dir: out

def mergeable(model, adapter_name_or_path) -> bool:
    return adapter_name_or_path is None or getattr(model, "quantization_method", None) is None

When it happens

Trigger: Export config where the model is loaded quantized (quantization_bit set, or model_name_or_path points at a 4/8-bit or GPTQ checkpoint) while adapter_name_or_path is also set.

Common situations: Fine-tuning on top of a pre-quantized community checkpoint (GPTQ/AWQ) and then trying to export a merged model.

Related errors

• Please merge adapters before quantizing the model.
• Quantization dataset is necessary for exporting.
• vLLM only accepts a single adapter. Merge them first.
• DoRA is not compatible with PTQ-quantized models.
• Cannot find satisfying example, considering decrease `export

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/3f7eb96ecd7fed70. Report an issue: GitHub.