vllm-project/vllm · error · ValueError

{model_config.dtype} is not supported for quantization metho

Error message

{model_config.dtype} is not supported for quantization method {model_config.quantization}. Supported dtypes: {supported_dtypes}

What it means

After the capability check, vLLM verifies model_config.dtype against quant_config.get_supported_act_dtypes(). Most quantization schemes only support a fixed set of activation dtypes (commonly float16 and bfloat16); requesting float32 or an unsupported half-precision variant with a quantized model raises this error. The dtype here is the engine activation dtype (--dtype), not the checkpoint's storage dtype.

Source

Thrown at vllm/config/vllm.py:782

        if model_config.quantization is not None:
            from vllm.model_executor.model_loader.weight_utils import get_quant_config

            quant_config = get_quant_config(model_config, load_config)
            capability_tuple = current_platform.get_device_capability()

            if capability_tuple is not None:
                capability = capability_tuple.to_int()
                if capability < quant_config.get_min_capability():
                    raise ValueError(
                        f"The quantization method {model_config.quantization} "
                        "is not supported for the current GPU. Minimum "
                        f"capability: {quant_config.get_min_capability()}. "
                        f"Current capability: {capability}."
                    )
            supported_dtypes = quant_config.get_supported_act_dtypes()
            if model_config.dtype not in supported_dtypes:
                raise ValueError(
                    f"{model_config.dtype} is not supported for quantization "
                    f"method {model_config.quantization}. Supported dtypes: "
                    f"{supported_dtypes}"
                )
            quant_config.maybe_update_config(
                model_config.model,
                hf_config=model_config.hf_config,
                revision=model_config.revision,
            )
            return quant_config
        return None

    @staticmethod
    def get_quantization_config(
        model_config: ModelConfig, load_config: LoadConfig
    ) -> QuantizationConfig | None:
        import copy

View on GitHub (pinned to c794754062)

Solutions

Use --dtype bfloat16 or --dtype float16 (or omit --dtype and let auto-detection pick from the checkpoint)
If you genuinely need float32 precision, serve the unquantized weights instead of a quantized checkpoint
Inspect quant_config.get_supported_act_dtypes() for your method (or its docs) before choosing --dtype

Example fix

# before
llm = LLM(model="...-fp8", dtype="float32")
# after
llm = LLM(model="...-fp8", dtype="bfloat16")

Defensive patterns

Strategy: validation

Validate before calling

import torch
SUPPORTED = {torch.float16, torch.bfloat16}  # typical; verify per method
dtype = torch.float32 if user_dtype == "float32" else torch.bfloat16
assert dtype in SUPPORTED, "quantized models require float16/bfloat16 activations"

Type guard

def dtype_supported(dtype, supported) -> bool:
    return dtype in supported

Prevention

Never combine --dtype float32 with quantized checkpoints
Omit --dtype for quantized models and let vLLM auto-select bf16/fp16
Read get_supported_act_dtypes() from the checkpoint's quant_config.json when unsure

When it happens

Trigger: Launching a quantized model with --dtype float32 or --dtype auto resolving to an unsupported dtype; e.g. an FP8/GPTQ checkpoint (supported_act_dtypes = {torch.float16, torch.bfloat16}) with --dtype float32; also custom quant configs that only list bfloat16 on platforms where users force float16.

Common situations: Setting --dtype float32 hoping for 'higher accuracy' on a quantized model; platforms (or CPU fallbacks) defaulting to float32; forcing float16 on a bf16-only quantized build; mixing --dtype with quantization overrides like quantization='fp8'.

Related errors

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/34fe5fee634bd321. Report an issue: GitHub.