huggingface/transformers · error · ValueError

Unsupported quantization method: '{self.quantization}'. Must

Error message

Unsupported quantization method: '{self.quantization}'. Must be 'bnb-4bit' or 'bnb-8bit'.

What it means

ModelManager._validate_args validates the --quantization option against the only two supported schemes: 'bnb-4bit' and 'bnb-8bit' (bitsandbytes). Any other non-None value — '4bit', 'int8', 'gptq', 'awq', 'none' — raises ValueError at manager construction time, before any model is loaded.

Source

Thrown at src/transformers/cli/serving/model_manager.py:189

        from ...utils.import_utils import is_kernels_available

        is_mps_device = (
            isinstance(device, str)
            and device.startswith("mps")
            or (device == "auto" and torch.backends.mps.is_available() and not torch.cuda.is_available())
        )
        if is_mps_device and is_kernels_available():
            logger.warning_once(
                "MPS detected and `kernels` is installed: defaulting attention to "
                "`kernels-community/metal-flash-sdpa@223ca3350d7ba32ecf19341ff2cbb8c43fa47d62. "
                "Pass `--attn-implementation sdpa` to opt out."
            )
            return "kernels-community/metal-flash-sdpa@223ca3350d7ba32ecf19341ff2cbb8c43fa47d62"
        return attn_implementation

    def _validate_args(self):
        if self.quantization is not None and self.quantization not in ("bnb-4bit", "bnb-8bit"):
            raise ValueError(
                f"Unsupported quantization method: '{self.quantization}'. Must be 'bnb-4bit' or 'bnb-8bit'."
            )
        VALID_ATTN_IMPLEMENTATIONS = {"eager", "sdpa", "flash_attention_2", "flash_attention_3", "flex_attention"}
        is_kernels_community = self.attn_implementation is not None and self.attn_implementation.startswith(
            "kernels-community/"
        )
        if (
            self.attn_implementation is not None
            and not is_kernels_community
            and self.attn_implementation not in VALID_ATTN_IMPLEMENTATIONS
        ):
            raise ValueError(
                f"Unsupported attention implementation: '{self.attn_implementation}'. "
                f"Must be one of {VALID_ATTN_IMPLEMENTATIONS} or a kernels-community kernel (e.g. 'kernels-community/flash-attn2')."
            )

    @staticmethod
    def process_model_name(model_id: str) -> str:

View on GitHub (pinned to a597f97485)

Solutions

Use --quantization bnb-4bit or --quantization bnb-8bit
For no quantization, omit the --quantization flag entirely
For other backends (gptq/awq), load the model yourself with from_pretrained and serve it via a custom app
Install bitsandbytes, which the bnb options require at load time

Example fix

# before
transformers serve --model_id llama --quantization 4bit

# after
transformers serve --model_id llama --quantization bnb-4bit

Defensive patterns

Strategy: validation

Validate before calling

if quantization is not None and quantization not in ("bnb-4bit", "bnb-8bit"):
    raise SystemExit(f"Unsupported quantization {quantization!r}; use bnb-4bit or bnb-8bit")

Type guard

def is_supported_quant(value: str | None) -> bool:
    return value is None or value in {"bnb-4bit", "bnb-8bit"}

Prevention

Restrict flags to the two bnb values or omit --quantization
For gptq/awq, plan a custom serving path instead of the CLI
Normalize common shorthands (4bit->bnb-4bit) in wrapper scripts

When it happens

Trigger: Passing --quantization gptq or awq (unsupported by the serve CLI even if transformers supports them via from_pretrained kwargs); shorthand '4bit'; passing 'none' as a string instead of omitting the flag; quoting issues that leave stray characters.

Common situations: Users assuming the serve CLI exposes all transformers quantization backends; porting flags from other servers (llama.cpp style 'q4_0', vLLM 'awq'); shell-quoting mistakes.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/3e632206247206bb. Report an issue: GitHub.