{"record":{"id":"3e632206247206bb","repo":"huggingface/transformers","slug":"unsupported-quantization-method-self-quantizati","errorCode":null,"errorMessage":"Unsupported quantization method: '{self.quantization}'. Must be 'bnb-4bit' or 'bnb-8bit'.","messagePattern":"Unsupported quantization method: '(.+?)'\\. Must be 'bnb-4bit' or 'bnb-8bit'\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/cli/serving/model_manager.py","lineNumber":189,"sourceCode":"        from ...utils.import_utils import is_kernels_available\n\n        is_mps_device = (\n            isinstance(device, str)\n            and device.startswith(\"mps\")\n            or (device == \"auto\" and torch.backends.mps.is_available() and not torch.cuda.is_available())\n        )\n        if is_mps_device and is_kernels_available():\n            logger.warning_once(\n                \"MPS detected and `kernels` is installed: defaulting attention to \"\n                \"`kernels-community/metal-flash-sdpa@223ca3350d7ba32ecf19341ff2cbb8c43fa47d62. \"\n                \"Pass `--attn-implementation sdpa` to opt out.\"\n            )\n            return \"kernels-community/metal-flash-sdpa@223ca3350d7ba32ecf19341ff2cbb8c43fa47d62\"\n        return attn_implementation\n\n    def _validate_args(self):\n        if self.quantization is not None and self.quantization not in (\"bnb-4bit\", \"bnb-8bit\"):\n            raise ValueError(\n                f\"Unsupported quantization method: '{self.quantization}'. Must be 'bnb-4bit' or 'bnb-8bit'.\"\n            )\n        VALID_ATTN_IMPLEMENTATIONS = {\"eager\", \"sdpa\", \"flash_attention_2\", \"flash_attention_3\", \"flex_attention\"}\n        is_kernels_community = self.attn_implementation is not None and self.attn_implementation.startswith(\n            \"kernels-community/\"\n        )\n        if (\n            self.attn_implementation is not None\n            and not is_kernels_community\n            and self.attn_implementation not in VALID_ATTN_IMPLEMENTATIONS\n        ):\n            raise ValueError(\n                f\"Unsupported attention implementation: '{self.attn_implementation}'. \"\n                f\"Must be one of {VALID_ATTN_IMPLEMENTATIONS} or a kernels-community kernel (e.g. 'kernels-community/flash-attn2').\"\n            )\n\n    @staticmethod\n    def process_model_name(model_id: str) -> str:","sourceCodeStart":171,"sourceCodeEnd":207,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/cli/serving/model_manager.py#L171-L207","documentation":"ModelManager._validate_args validates the --quantization option against the only two supported schemes: 'bnb-4bit' and 'bnb-8bit' (bitsandbytes). Any other non-None value — '4bit', 'int8', 'gptq', 'awq', 'none' — raises ValueError at manager construction time, before any model is loaded.","triggerScenarios":"Passing --quantization gptq or awq (unsupported by the serve CLI even if transformers supports them via from_pretrained kwargs); shorthand '4bit'; passing 'none' as a string instead of omitting the flag; quoting issues that leave stray characters.","commonSituations":"Users assuming the serve CLI exposes all transformers quantization backends; porting flags from other servers (llama.cpp style 'q4_0', vLLM 'awq'); shell-quoting mistakes.","solutions":["Use --quantization bnb-4bit or --quantization bnb-8bit","For no quantization, omit the --quantization flag entirely","For other backends (gptq/awq), load the model yourself with from_pretrained and serve it via a custom app","Install bitsandbytes, which the bnb options require at load time"],"exampleFix":"# before\ntransformers serve --model_id llama --quantization 4bit\n\n# after\ntransformers serve --model_id llama --quantization bnb-4bit","handlingStrategy":"validation","validationCode":"if quantization is not None and quantization not in (\"bnb-4bit\", \"bnb-8bit\"):\n    raise SystemExit(f\"Unsupported quantization {quantization!r}; use bnb-4bit or bnb-8bit\")","typeGuard":"def is_supported_quant(value: str | None) -> bool:\n    return value is None or value in {\"bnb-4bit\", \"bnb-8bit\"}","tryCatchPattern":null,"preventionTips":["Restrict flags to the two bnb values or omit --quantization","For gptq/awq, plan a custom serving path instead of the CLI","Normalize common shorthands (4bit->bnb-4bit) in wrapper scripts"],"tags":["serving","quantization","cli","validation","valueerror"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}