hiyouga/LlamaFactory · error · ValueError

vLLM engine does not support bnb quantization (GPTQ and AWQ

Error message

vLLM engine does not support bnb quantization (GPTQ and AWQ are supported).

What it means

Raised in get_infer_args when infer_backend is vllm and model_args.quantization_bit is not None. The quantization_bit field selects bitsandbytes (bnb) PTQ quantization, which only works inside the transformers/HF loading path; vLLM has its own quantization formats (GPTQ, AWQ) configured differently.

Source

Thrown at src/llamafactory/hparams/parser.py:681

            _get_kt_runtime_capacity(data_args, training_args, finetuning_args),
        )

    return model_args, data_args, training_args, finetuning_args, generating_args


def get_infer_args(args: dict[str, Any] | list[str] | None = None) -> _INFER_CLS:
    model_args, data_args, finetuning_args, generating_args = _parse_infer_args(args)

    # Setup logging
    _set_transformers_logging()

    # Check arguments
    if model_args.infer_backend == "vllm":
        if finetuning_args.stage != "sft":
            raise ValueError("vLLM engine only supports auto-regressive models.")

        if model_args.quantization_bit is not None:
            raise ValueError("vLLM engine does not support bnb quantization (GPTQ and AWQ are supported).")

        if model_args.rope_scaling is not None:
            raise ValueError("vLLM engine does not support RoPE scaling.")

        if model_args.adapter_name_or_path is not None and len(model_args.adapter_name_or_path) != 1:
            raise ValueError("vLLM only accepts a single adapter. Merge them first.")

    _set_env_vars()
    _verify_model_args(model_args, data_args, finetuning_args)
    _check_extra_dependencies(model_args, finetuning_args)

    # Post-process model arguments
    if model_args.export_dir is not None and model_args.export_device == "cpu":
        model_args.device_map = {"": torch.device("cpu")}
        if data_args.cutoff_len != DataArguments().cutoff_len:  # override cutoff_len if it is not default
            model_args.model_max_length = data_args.cutoff_len
    else:
        model_args.device_map = "auto"

View on GitHub (pinned to f28afaf635)

Solutions

Remove quantization_bit from the model arguments when using vLLM.
If you need a 4-bit model under vLLM, download/use a GPTQ- or AWQ-quantized checkpoint (e.g. models ending in -GPTQ-Int4 or -AWQ) and let vLLM load it natively.
Switch infer_backend to hf if you must keep bnb quantization.

Example fix

# before
model_name_or_path: meta-llama/Llama-3-8B
infer_backend: vllm
quantization_bit: 4

# after
model_name_or_path: Qwen/Qwen2-7B-Instruct-GPTQ-Int4
infer_backend: vllm

Defensive patterns

Strategy: validation

Validate before calling

if cfg["model_args"].get("infer_backend") == "vllm":
    assert cfg["model_args"].get("quantization_bit") is None, \
        "bnb quantization_bit is HF-only; use a GPTQ/AWQ checkpoint for vLLM"

Prevention

Maintain separate model presets per backend: bnb bits for HF, pre-quantized GPTQ/AWQ repos for vLLM.
Never copy the training quantization block into a serving config.

When it happens

Trigger: A chat/inference config with infer_backend: vllm plus quantization_bit: 8 or 4 in the model section.

Common situations: Copy-pasting the quantization settings from a QLoRA training config into a vLLM serving config; assuming the same quantization knob works for both backends.

Related errors

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/85a88992d2b5abde. Report an issue: GitHub.