{"record":{"id":"34fe5fee634bd321","repo":"vllm-project/vllm","slug":"model-config-dtype-is-not-supported-for-quantiza","errorCode":null,"errorMessage":"{model_config.dtype} is not supported for quantization method {model_config.quantization}. Supported dtypes: {supported_dtypes}","messagePattern":"(.+?) is not supported for quantization method (.+?)\\. Supported dtypes: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"vllm/config/vllm.py","lineNumber":782,"sourceCode":"\n        if model_config.quantization is not None:\n            from vllm.model_executor.model_loader.weight_utils import get_quant_config\n\n            quant_config = get_quant_config(model_config, load_config)\n            capability_tuple = current_platform.get_device_capability()\n\n            if capability_tuple is not None:\n                capability = capability_tuple.to_int()\n                if capability < quant_config.get_min_capability():\n                    raise ValueError(\n                        f\"The quantization method {model_config.quantization} \"\n                        \"is not supported for the current GPU. Minimum \"\n                        f\"capability: {quant_config.get_min_capability()}. \"\n                        f\"Current capability: {capability}.\"\n                    )\n            supported_dtypes = quant_config.get_supported_act_dtypes()\n            if model_config.dtype not in supported_dtypes:\n                raise ValueError(\n                    f\"{model_config.dtype} is not supported for quantization \"\n                    f\"method {model_config.quantization}. Supported dtypes: \"\n                    f\"{supported_dtypes}\"\n                )\n            quant_config.maybe_update_config(\n                model_config.model,\n                hf_config=model_config.hf_config,\n                revision=model_config.revision,\n            )\n            return quant_config\n        return None\n\n    @staticmethod\n    def get_quantization_config(\n        model_config: ModelConfig, load_config: LoadConfig\n    ) -> QuantizationConfig | None:\n        import copy\n","sourceCodeStart":764,"sourceCodeEnd":800,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/config/vllm.py#L764-L800","documentation":"After the capability check, vLLM verifies model_config.dtype against quant_config.get_supported_act_dtypes(). Most quantization schemes only support a fixed set of activation dtypes (commonly float16 and bfloat16); requesting float32 or an unsupported half-precision variant with a quantized model raises this error. The dtype here is the engine activation dtype (--dtype), not the checkpoint's storage dtype.","triggerScenarios":"Launching a quantized model with --dtype float32 or --dtype auto resolving to an unsupported dtype; e.g. an FP8/GPTQ checkpoint (supported_act_dtypes = {torch.float16, torch.bfloat16}) with --dtype float32; also custom quant configs that only list bfloat16 on platforms where users force float16.","commonSituations":"Setting --dtype float32 hoping for 'higher accuracy' on a quantized model; platforms (or CPU fallbacks) defaulting to float32; forcing float16 on a bf16-only quantized build; mixing --dtype with quantization overrides like quantization='fp8'.","solutions":["Use --dtype bfloat16 or --dtype float16 (or omit --dtype and let auto-detection pick from the checkpoint)","If you genuinely need float32 precision, serve the unquantized weights instead of a quantized checkpoint","Inspect quant_config.get_supported_act_dtypes() for your method (or its docs) before choosing --dtype"],"exampleFix":"# before\nllm = LLM(model=\"...-fp8\", dtype=\"float32\")\n# after\nllm = LLM(model=\"...-fp8\", dtype=\"bfloat16\")","handlingStrategy":"validation","validationCode":"import torch\nSUPPORTED = {torch.float16, torch.bfloat16}  # typical; verify per method\ndtype = torch.float32 if user_dtype == \"float32\" else torch.bfloat16\nassert dtype in SUPPORTED, \"quantized models require float16/bfloat16 activations\"","typeGuard":"def dtype_supported(dtype, supported) -> bool:\n    return dtype in supported","tryCatchPattern":null,"preventionTips":["Never combine --dtype float32 with quantized checkpoints","Omit --dtype for quantized models and let vLLM auto-select bf16/fp16","Read get_supported_act_dtypes() from the checkpoint's quant_config.json when unsure"],"tags":["quantization","dtype","config","validation"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}