vllm-project/vllm · error · ValueError

The model type {model_type!r} does not support float16. Reas

Error message

The model type {model_type!r} does not support float16. Reason: {reason}

What it means

_check_valid_dtype raises when the model type appears in _FLOAT16_NOT_SUPPORTED_MODELS (with a per-model reason) and float16 was requested. Certain architectures (numerically unstable in fp16, e.g. some Gemma/Nemotron variants) are hard-blocked from float16 regardless of hardware.

Source

Thrown at vllm/config/model.py:2205

_FLOAT16_NOT_SUPPORTED_MODELS = {
    "gemma2": "Numerical instability. Please use bfloat16 or float32 instead.",
    "gemma3": "Numerical instability. Please use bfloat16 or float32 instead.",
    "gemma3_text": "Numerical instability. Please use bfloat16 or float32 instead.",
    "glm4": "Numerical instability. Please use bfloat16 or float32 instead.",
}


def _is_valid_dtype(model_type: str, dtype: torch.dtype):
    if model_type in _FLOAT16_NOT_SUPPORTED_MODELS and dtype == torch.float16:  # noqa: E501, SIM103
        return False

    return True


def _check_valid_dtype(model_type: str, dtype: torch.dtype):
    if model_type in _FLOAT16_NOT_SUPPORTED_MODELS and dtype == torch.float16:
        reason = _FLOAT16_NOT_SUPPORTED_MODELS[model_type]
        raise ValueError(
            f"The model type {model_type!r} does not support float16. Reason: {reason}"
        )

    return True


def _resolve_auto_dtype(
    model_type: str,
    config_dtype: torch.dtype,
    *,
    is_pooling_model: bool,
):
    supported_dtypes = [
        dtype
        for dtype in current_platform.supported_dtypes
        if _is_valid_dtype(model_type, dtype)
    ]

View on GitHub (pinned to c794754062)

Solutions

Use bfloat16 (dtype='bfloat16') or auto for the model.
Use a variant of the model trained/tuned for fp16 if the vendor provides one.
If config.json sets torch_dtype: float16, override at launch with --dtype bfloat16.

Example fix

# before
vllm serve google/gemma-2-9b --dtype float16
# after
vllm serve google/gemma-2-9b --dtype bfloat16

Defensive patterns

Strategy: validation

Validate before calling

from vllm.config.model import _FLOAT16_NOT_SUPPORTED_MODELS
def dtype_allowed(model_type: str, dtype) -> bool:
    return not (model_type in _FLOAT16_NOT_SUPPORTED_MODELS and str(dtype).endswith('float16'))
# default to bfloat16/auto for listed model types

Prevention

Default to --dtype auto; only pin float16 on hardware without bf16 and for models known to be fp16-stable.
Check _FLOAT16_NOT_SUPPORTED_MODELS (or release notes) when adding new architectures to your fleet.
Fix checkpoints whose config.json hardcodes torch_dtype: float16 for blocked model types.

When it happens

Trigger: Passing dtype='float16' (or a config whose torch_dtype resolves to fp16) for a model_type listed in _FLOAT16_NOT_SUPPORTED_MODELS; raised from _get_and_verify_dtype during ModelConfig construction.

Common situations: Forcing fp16 on V100-class GPUs (no bf16) for a model family that overflows in fp16; fine-tuned/community checkpoints that defaulted torch_dtype to float16 in config.json.

Related errors

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/85caa527ee4ee24e. Report an issue: GitHub.