{"record":{"id":"85a88992d2b5abde","repo":"hiyouga/LlamaFactory","slug":"vllm-engine-does-not-support-bnb-quantization-gpt","errorCode":null,"errorMessage":"vLLM engine does not support bnb quantization (GPTQ and AWQ are supported).","messagePattern":"vLLM engine does not support bnb quantization \\(GPTQ and AWQ are supported\\)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/parser.py","lineNumber":681,"sourceCode":"            _get_kt_runtime_capacity(data_args, training_args, finetuning_args),\n        )\n\n    return model_args, data_args, training_args, finetuning_args, generating_args\n\n\ndef get_infer_args(args: dict[str, Any] | list[str] | None = None) -> _INFER_CLS:\n    model_args, data_args, finetuning_args, generating_args = _parse_infer_args(args)\n\n    # Setup logging\n    _set_transformers_logging()\n\n    # Check arguments\n    if model_args.infer_backend == \"vllm\":\n        if finetuning_args.stage != \"sft\":\n            raise ValueError(\"vLLM engine only supports auto-regressive models.\")\n\n        if model_args.quantization_bit is not None:\n            raise ValueError(\"vLLM engine does not support bnb quantization (GPTQ and AWQ are supported).\")\n\n        if model_args.rope_scaling is not None:\n            raise ValueError(\"vLLM engine does not support RoPE scaling.\")\n\n        if model_args.adapter_name_or_path is not None and len(model_args.adapter_name_or_path) != 1:\n            raise ValueError(\"vLLM only accepts a single adapter. Merge them first.\")\n\n    _set_env_vars()\n    _verify_model_args(model_args, data_args, finetuning_args)\n    _check_extra_dependencies(model_args, finetuning_args)\n\n    # Post-process model arguments\n    if model_args.export_dir is not None and model_args.export_device == \"cpu\":\n        model_args.device_map = {\"\": torch.device(\"cpu\")}\n        if data_args.cutoff_len != DataArguments().cutoff_len:  # override cutoff_len if it is not default\n            model_args.model_max_length = data_args.cutoff_len\n    else:\n        model_args.device_map = \"auto\"","sourceCodeStart":663,"sourceCodeEnd":699,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/parser.py#L663-L699","documentation":"Raised in get_infer_args when infer_backend is vllm and model_args.quantization_bit is not None. The quantization_bit field selects bitsandbytes (bnb) PTQ quantization, which only works inside the transformers/HF loading path; vLLM has its own quantization formats (GPTQ, AWQ) configured differently.","triggerScenarios":"A chat/inference config with infer_backend: vllm plus quantization_bit: 8 or 4 in the model section.","commonSituations":"Copy-pasting the quantization settings from a QLoRA training config into a vLLM serving config; assuming the same quantization knob works for both backends.","solutions":["Remove quantization_bit from the model arguments when using vLLM.","If you need a 4-bit model under vLLM, download/use a GPTQ- or AWQ-quantized checkpoint (e.g. models ending in -GPTQ-Int4 or -AWQ) and let vLLM load it natively.","Switch infer_backend to hf if you must keep bnb quantization."],"exampleFix":"# before\nmodel_name_or_path: meta-llama/Llama-3-8B\ninfer_backend: vllm\nquantization_bit: 4\n\n# after\nmodel_name_or_path: Qwen/Qwen2-7B-Instruct-GPTQ-Int4\ninfer_backend: vllm","handlingStrategy":"validation","validationCode":"if cfg[\"model_args\"].get(\"infer_backend\") == \"vllm\":\n    assert cfg[\"model_args\"].get(\"quantization_bit\") is None, \\\n        \"bnb quantization_bit is HF-only; use a GPTQ/AWQ checkpoint for vLLM\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Maintain separate model presets per backend: bnb bits for HF, pre-quantized GPTQ/AWQ repos for vLLM.","Never copy the training quantization block into a serving config."],"tags":["vllm","bnb","quantization","gptq","awq"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}