vllm-project/vllm · error · ValueError

The quantization method {model_config.quantization} is not s

Error message

The quantization method {model_config.quantization} is not supported for the current GPU. Minimum capability: {quant_config.get_min_capability()}. Current capability: {capability}.

What it means

During quantization config resolution, vLLM fetches the model's quant config, queries the current platform's device capability, and compares it against quant_config.get_min_capability(). If the GPU's compute capability (as an int, e.g. 80 for SM80, 90 for SM90) is below the quantization method's minimum, it raises this error. For example, FP8 or certain AWQ/Marlin variants require SM80+ or SM89+, and running them on older GPUs produces undefined kernels, so vLLM refuses to start.

Source

Thrown at vllm/config/vllm.py:774

            enable_trace_function_call(log_path)

    @staticmethod
    def _get_quantization_config(
        model_config: ModelConfig, load_config: LoadConfig
    ) -> QuantizationConfig | None:
        """Get the quantization config."""
        from vllm.platforms import current_platform

        if model_config.quantization is not None:
            from vllm.model_executor.model_loader.weight_utils import get_quant_config

            quant_config = get_quant_config(model_config, load_config)
            capability_tuple = current_platform.get_device_capability()

            if capability_tuple is not None:
                capability = capability_tuple.to_int()
                if capability < quant_config.get_min_capability():
                    raise ValueError(
                        f"The quantization method {model_config.quantization} "
                        "is not supported for the current GPU. Minimum "
                        f"capability: {quant_config.get_min_capability()}. "
                        f"Current capability: {capability}."
                    )
            supported_dtypes = quant_config.get_supported_act_dtypes()
            if model_config.dtype not in supported_dtypes:
                raise ValueError(
                    f"{model_config.dtype} is not supported for quantization "
                    f"method {model_config.quantization}. Supported dtypes: "
                    f"{supported_dtypes}"
                )
            quant_config.maybe_update_config(
                model_config.model,
                hf_config=model_config.hf_config,
                revision=model_config.revision,
            )
            return quant_config

View on GitHub (pinned to c794754062)

Solutions

Use a checkpoint quantized for your GPU class (e.g. GPTQ/AWQ int4 for pre-FP8 GPUs) or an unquantized BF16/FP16 checkpoint
Run on hardware meeting the minimum capability stated in the message (e.g. H100/Ada for FP8 methods requiring >=89/90)
Check current capability first: torch.cuda.get_device_capability() and compare against the method's documented minimum before launching
Set model_config.quantization to a method supported by your card or leave auto-detection to pick a compatible kernel

Example fix

# before
llm = LLM(model="red-panda/...-fp8")  # on RTX 3090 (SM86 < min 89)
# after
llm = LLM(model="...-awq-int4")  # AWQ works on SM86
# or run on H100/Ada (SM89/90+)

Defensive patterns

Strategy: validation

Validate before calling

import torch
cap = torch.cuda.get_device_capability(0)
cap_int = cap[0] * 10 + cap[1]
MIN_CAP = {"fp8": 89, "awq_marlin": 80}  # per method; consult quant_config
assert cap_int >= MIN_CAP.get(quant_method, 0), \
    f"GPU capability {cap_int} below minimum for {quant_method}"

Type guard

def gpu_supports(cap_int: int, min_cap: int) -> bool:
    return cap_int >= min_cap

Try / catch

try:
    engine = LLM(model=ckpt)
except ValueError as e:
    if "Minimum capability" in str(e):
        raise SystemExit(f"GPU too old for {ckpt}; pick a GPTQ/AWQ or bf16 checkpoint")
    raise

Prevention

Check torch.cuda.get_device_capability() against the quant method's minimum before pulling large checkpoints
Prefer GPTQ/AWQ int4 checkpoints for pre-Ampere or Ampere consumer cards
Automate checkpoint selection per node GPU class in deployment tooling

When it happens

Trigger: Loading a quantized checkpoint (e.g. compressed-tensors FP8, AWQ marlin, GPTQ marlin) on a GPU whose capability is below the method's minimum: FP8 checkpoints on pre-Ampere cards, some FP8 variants on Ampere (SM80/86) when min is 89, etc. Raised from get_quant_config path inside VllmConfig when model_config.quantization is set and current_platform.get_device_capability() returns a tuple.

Common situations: Downloading an FP8-quantized community model (most are FP8 w8a8) and running it on a 3090/A10/A100 that lacks required capability; migrating between GPU generations; using quantization='fp8' explicitly on T4/V100 hardware; headless CI machines with older GPUs.

Related errors

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/f368aa1bfcc809fe. Report an issue: GitHub.