{"record":{"id":"c90454e06282f12f","repo":"sgl-project/sglang","slug":"sglang-s-autoround-cpu-inference-path-currently-su","errorCode":null,"errorMessage":"SGLang's AutoRound CPU inference path currently supports only 4-bit AWQ/GPTQ checkpoints because it uses the Intel AMX INT4 backend, but got {weight_bits}-bit.","messagePattern":"SGLang's AutoRound CPU inference path currently supports only 4-bit AWQ/GPTQ checkpoints because it uses the Intel AMX INT4 backend, but got (.+?)-bit\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/auto_round.py","lineNumber":283,"sourceCode":"                        layer_name.replace(fusion_key, sub_key) for sub_key in sub_keys\n                    ]\n                    sub_configs = [get_config(name, quantized) for name in sub_names]\n                    if len(set(sub_configs)) == 1:\n                        return sub_configs[0]\n                    raise ValueError(\n                        f\"Fused module '{layer_name}' requires \"\n                        f\"consistent quant config for {sub_names}\"\n                    )\n\n        # 5. Fallback or try a regular expression match\n        return get_config(layer_name, quantized)\n\n    def check_quantized(self, weight_bits: int) -> bool:\n        return weight_bits < 16\n\n    def check_cpu_support(self, weight_bits: int) -> None:\n        if weight_bits != 4:\n            raise ValueError(\n                \"SGLang's AutoRound CPU inference path currently supports \"\n                \"only 4-bit AWQ/GPTQ checkpoints because it uses the Intel \"\n                f\"AMX INT4 backend, but got {weight_bits}-bit.\"\n            )\n        if not _is_cpu_amx_available:\n            raise ValueError(_CPU_AMX_REQUIRED_MSG)\n\n    def log_gptq_default_assumptions_once(self) -> None:\n        if self._logged_gptq_default_assumptions or not self.gptq_defaulted_config_keys:\n            return\n        self._logged_gptq_default_assumptions = True\n        default_summary = {\n            key: _GPTQ_DEFAULTS[key] for key in self.gptq_defaulted_config_keys\n        }\n        logger.info(\n            \"AutoRound GPTQ config does not specify %s; using SGLang defaults %s.\",\n            \", \".join(self.gptq_defaulted_config_keys),\n            default_summary,","sourceCodeStart":265,"sourceCodeEnd":301,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/auto_round.py#L265-L301","documentation":"check_cpu_support gates SGLang's AutoRound CPU path to 4-bit checkpoints because it is implemented on the Intel AMX INT4 backend. Any other bit-width (e.g. 8-bit) on CPU raises before loading.","triggerScenarios":"Serving an 8-bit (or other) AutoRound AWQ/GPTQ checkpoint with --quantization auto_round on a CPU/AMX machine; apply_awq_quant_layer/apply_gptq_quant_layer call check_cpu_support during weight loading.","commonSituations":"Reusing a GPU-tuned 8-bit AutoRound model on CPU servers; Intel Xeon AMX deployments assuming W8A8 works for AutoRound.","solutions":["Use a 4-bit AutoRound checkpoint for CPU inference","Serve the 8-bit model on GPU, or use a CPU-supported w8a8 path if available for the quant method","Re-export from AutoRound with 4-bit weights"],"exampleFix":"# before\n--quantization auto_round   # 8-bit ckpt on CPU\n# after\n--quantization auto_round   # with 4-bit re-exported ckpt","handlingStrategy":"type-guard","validationCode":"from sglang.srt.utils import is_cpu\nif is_cpu():\n    assert quant_cfg[\"weight_bits\"] == 4, \"CPU AutoRound requires 4-bit\"","typeGuard":"def cpu_autoround_ok(weight_bits: int, is_cpu: bool) -> bool:\n    return (not is_cpu) or weight_bits == 4","tryCatchPattern":"try:\n    method.create_weights(...)  # or launch server\nexcept ValueError as e:\n    if \"AMX INT4\" in str(e): switch_to_4bit_checkpoint_or_gpu()\n    raise","preventionTips":["Maintain separate 4-bit checkpoints for CPU AMX serving","Gate model selection on detected hardware in deployment scripts"],"tags":["quantization","auto-round","cpu","amx","weight-bits"],"backgroundTag":"unsupported-platform-feature","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}