{"record":{"id":"830667361156c5d6","repo":"sgl-project/sglang","slug":"currently-only-4bits-is-supported-on-cpu-with-amx","errorCode":null,"errorMessage":"Currently, only 4bits is supported on CPU with AMX.","messagePattern":"Currently, only 4bits is supported on CPU with AMX\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/gptq/schemes/gptq_cpu.py","lineNumber":41,"sourceCode":"from .gptq_scheme import GPTQMoESchemeBase\n\nif TYPE_CHECKING:\n    from sglang.srt.layers.moe.token_dispatcher import StandardDispatchOutput\n    from sglang.srt.layers.quantization.gptq.gptq import GPTQConfig\n\n__all__ = [\"GPTQIntelAMXLinearScheme\", \"GPTQIntelAMXMoEScheme\"]\n\n\ndef _check_cpu_amx_support(quant_config: GPTQConfig) -> None:\n    if quant_config.desc_act and not (\n        quant_config.true_sequential and quant_config.static_groups\n    ):\n        raise ValueError(\n            \"Currently, desc_act (True) is only supported with sequential \"\n            \"and static group on CPU with AMX.\"\n        )\n    if quant_config.weight_bits != 4:\n        raise ValueError(\"Currently, only 4bits is supported on CPU with AMX.\")\n    if quant_config.checkpoint_format == \"gptq_v2\":\n        raise ValueError(\"Currently, gptq_v2 is not supported on CPU with AMX.\")\n\n\nclass GPTQIntelAMXLinearScheme(GPTQLinearScheme):\n    \"\"\"Linear scheme for GPTQ on Intel CPU with AMX.\"\"\"\n\n    def _init_kernel(self, quant_config: GPTQConfig):\n        return GPTQIntelAMXLinearKernel(quant_config)\n\n    def create_weights(\n        self,\n        layer: torch.nn.Module,\n        input_size_per_partition: int,\n        output_partition_sizes: list[int],\n        input_size: int,\n        params_dtype: torch.dtype,\n        weight_loader,","sourceCodeStart":23,"sourceCodeEnd":59,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/gptq/schemes/gptq_cpu.py#L23-L59","documentation":"GPTQ CPU/AMX kernel (GPTQIntelAMXLinearScheme) only supports 4-bit weights; the check in _check_cpu_amx_support rejects any quant_config.weight_bits other than 4 (e.g. 2-bit, 3-bit, 8-bit GPTQ checkpoints) before weight creation.","triggerScenarios":"Loading a model quantized with GPTQ wbits=2, 3, or 8 on an Intel CPU using the AMX path (e.g. quant_config with weight_bits != 4 while running with the Intel AMX scheme); raised from create_weights via _check_cpu_amx_support.","commonSituations":"Running a GPTQ-Int8 or GPTQ-2bit HuggingFace checkpoint on Xeon servers with AMX tiles enabled; switching a deployment from GPU to CPU without requantizing the model.","solutions":["Use a 4-bit GPTQ quantized checkpoint (wbits=4) for CPU AMX inference","Requantize the model to 4-bit GPTQ with static group size","Fall back to a different backend/device (e.g. GPU or a non-AMX CPU kernel) that supports the checkpoint's bit width"],"exampleFix":"# before\nmodel = AutoModelForCausalLM.from_pretrained(\"org/model-gptq-8bit\")  # weight_bits=8\n\n# after\nmodel = ...  # use \"org/model-gptq-4bit\" with quantization_config weight_bits=4","handlingStrategy":"validation","validationCode":"cfg = GPTQConfig.from_pretrained(model_path)\nif cfg.quant_method == \"gptq\" and is_cpu_amx_backend() and cfg.weight_bits != 4:\n    raise SystemExit(\"Need a 4-bit GPTQ checkpoint for CPU AMX; got \" f\"{cfg.weight_bits}-bit\")","typeGuard":"def is_amx_compatible_gptq(cfg: GPTQConfig) -> bool:\n    return cfg.weight_bits == 4 and cfg.checkpoint_format != \"gptq_v2\" and not (cfg.desc_act and not cfg.static_groups)","tryCatchPattern":null,"preventionTips":["Standardize on 4-bit GPTQ checkpoints for CPU deployments","Assert weight_bits and checkpoint_format in a preflight config check before loading weights"],"tags":["gptq","quantization","cpu","amx","unsupported-operation"],"backgroundTag":"unsupported-quantization-config","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}