What causes "Unknown dtype: {head_dtype!r}" in vllm-project/vllm?

Set head_dtype in config.json to a recognized name ('float16', 'bfloat16', 'float32') or to 'model' to inherit the body dtype.

vllm-project/vllm · error · ValueError

Unknown dtype: {head_dtype!r}

Error message

Unknown dtype: {head_dtype!r}

What it means

_get_head_dtype reads an optional head_dtype field from the HF config to give the lm_head/pooling head a different precision than the body. A string head_dtype that is not a key of _STR_DTYPE_TO_TORCH_DTYPE (after lowercasing) raises this ValueError.

Source

Thrown at vllm/config/model.py:2314

            logger.info("Downcasting %s to %s.", config_dtype, torch_dtype)
        else:
            # Casting between float16 and bfloat16 is allowed with a warning.
            logger.warning("Casting %s to %s.", config_dtype, torch_dtype)

    return torch_dtype


def _get_head_dtype(
    config: PretrainedConfig, dtype: torch.dtype, runner_type: str
) -> torch.dtype:
    head_dtype: str | torch.dtype | None = getattr(config, "head_dtype", None)

    if head_dtype == "model":
        return dtype
    elif isinstance(head_dtype, str):
        head_dtype = head_dtype.lower()
        if head_dtype not in _STR_DTYPE_TO_TORCH_DTYPE:
            raise ValueError(f"Unknown dtype: {head_dtype!r}")
        return _STR_DTYPE_TO_TORCH_DTYPE[head_dtype]
    elif isinstance(head_dtype, torch.dtype):
        return head_dtype
    elif head_dtype is None:
        if torch.float32 not in current_platform.supported_dtypes:
            return dtype
        if runner_type == "pooling":
            return torch.float32
        return dtype
    else:
        raise ValueError(f"Unknown dtype: {head_dtype}")


def _get_and_verify_max_len(
    hf_config: PretrainedConfig,
    model_arch_config: ModelArchitectureConfig,
    tokenizer_config: dict | None,
    max_model_len: int | None,

View on GitHub (pinned to c794754062)

Solutions

Set head_dtype in config.json to a recognized name ('float16', 'bfloat16', 'float32') or to 'model' to inherit the body dtype.
Remove the head_dtype field entirely to fall back to the default resolution (body dtype, or fp32 for pooling where supported).

Example fix

// before: config.json
"head_dtype": "fp16"
// after
"head_dtype": "float16"

Defensive patterns

Strategy: validation

Validate before calling

from vllm.config.model import _STR_DTYPE_TO_TORCH_DTYPE
def head_dtype_ok(hd) -> bool:
    return hd is None or hd == 'model' or (
        isinstance(hd, str) and hd.lower() in _STR_DTYPE_TO_TORCH_DTYPE)

Type guard

def is_valid_head_dtype(hd: object) -> bool:
    return hd is None or hd == 'model' or (
        isinstance(hd, str) and hd.lower() in _STR_DTYPE_TO_TORCH_DTYPE) or (
        isinstance(hd, __import__('torch').dtype))

Prevention

Only write recognized dtype names or 'model' into head_dtype in config.json.
Schema-validate checkpoint configs before deployment (head_dtype must be string or absent).
Prefer omitting head_dtype unless per-head precision is deliberately needed.

When it happens

Trigger: config.head_dtype is a string like 'fp16' or 'float' that is not in the dtype table; triggered whenever ModelConfig verifies head dtypes for a runner.

Common situations: Fine-tuned checkpoints saved with a nonstandard head_dtype value; hand-edited config.json; model authors experimenting with per-head precision fields that vLLM only partially recognizes.

Related errors

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/40aa00504b208b82. Report an issue: GitHub.