huggingface/transformers · error · ValueError

Unsupported dtype: '{dtype}'. Must be 'auto' or a valid torc

Error message

Unsupported dtype: '{dtype}'. Must be 'auto' or a valid torch dtype (e.g. 'float16', 'bfloat16').

What it means

ModelManager._resolve_dtype maps the --dtype CLI value onto a torch attribute via getattr(torch, dtype) and verifies the result is a torch.dtype instance. 'auto' and None pass through; anything that is not an attribute of torch or not a dtype object (e.g. 'fp16', 'float', 'int') raises ValueError.

Source

Thrown at src/transformers/cli/serving/model_manager.py:156

        self._validate_args()

        # Preloaded models should never be auto-unloaded
        if force_model is not None:
            self.model_timeout = -1

        # Preload the forced model after all state is initialized
        if force_model is not None:
            self.load_model_and_processor(self.process_model_name(force_model))

    @staticmethod
    def _resolve_dtype(dtype: str | None):
        import torch

        if dtype in ("auto", None):
            return dtype
        resolved = getattr(torch, dtype, None)
        if not isinstance(resolved, torch.dtype):
            raise ValueError(
                f"Unsupported dtype: '{dtype}'. Must be 'auto' or a valid torch dtype (e.g. 'float16', 'bfloat16')."
            )
        return resolved

    @classmethod
    def _resolve_attn_implementation(cls, attn_implementation: str | None, device: str | int) -> str | None:
        r"""
        Default to a fast kernel for `mps` when available.
        """
        if attn_implementation is not None:
            return attn_implementation

        import torch

        from ...utils.import_utils import is_kernels_available

        is_mps_device = (
            isinstance(device, str)

View on GitHub (pinned to a597f97485)

Solutions

1. Use exact torch dtype names: --dtype float16, --dtype bfloat16, --dtype float32, or --dtype auto
2. Spell abbreviations out (fp16 -> float16, bf16 -> bfloat16)
3. Strip whitespace/casing issues: values are case-sensitive attribute names on torch

Example fix

# before
transformers serve --model_id gpt2 --dtype fp16   # ValueError

# after
transformers serve --model_id gpt2 --dtype float16

import torch

DTYPES = {"auto"} | {k for k, v in vars(torch).items() if isinstance(v, torch.dtype)}
if dtype not in DTYPES:
    raise SystemExit(f"Bad dtype {dtype!r}; choose from {sorted(DTYPES)}")

import torch

def is_valid_dtype(value: str) -> bool:
    if value in ("auto", None):
        return True
    return isinstance(getattr(torch, value, None), torch.dtype)

When it happens

Trigger: Passing --dtype fp16 (abbreviation instead of full name); --dtype float or 'half'; a value like 'auto ' with trailing whitespace; any string that resolves to a non-dtype torch attribute (e.g. 'float' resolves to the builtin-like torch.float which is a dtype, but 'float32_min_normal'-style names would not be).

Common situations: Habitual fp16/bf16 shorthand from other frameworks; config files or scripts written for vLLM-style flags; copy-pasting dtype strings from docs of different tools.

Related errors

• Unsupported quantization method: '{self.quantization}'. Must
• Unsupported attention implementation: '{self.attn_implementa
• `axis_value` for `HQQ` backend has to be one of [`0`, `1`] b
• You can construct a Cache either from a list `layers` of all
• You should provide exactly one of `layers` or `layer_class_t

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/042b7bd28c37bf60. Report an issue: GitHub.