{"record":{"id":"15908b1f63f00e5c","repo":"sgl-project/sglang","slug":"currently-gptq-v2-is-not-supported-on-cpu-with-am","errorCode":null,"errorMessage":"Currently, gptq_v2 is not supported on CPU with AMX.","messagePattern":"Currently, gptq_v2 is not supported on CPU with AMX\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/gptq/schemes/gptq_cpu.py","lineNumber":43,"sourceCode":"if TYPE_CHECKING:\n    from sglang.srt.layers.moe.token_dispatcher import StandardDispatchOutput\n    from sglang.srt.layers.quantization.gptq.gptq import GPTQConfig\n\n__all__ = [\"GPTQIntelAMXLinearScheme\", \"GPTQIntelAMXMoEScheme\"]\n\n\ndef _check_cpu_amx_support(quant_config: GPTQConfig) -> None:\n    if quant_config.desc_act and not (\n        quant_config.true_sequential and quant_config.static_groups\n    ):\n        raise ValueError(\n            \"Currently, desc_act (True) is only supported with sequential \"\n            \"and static group on CPU with AMX.\"\n        )\n    if quant_config.weight_bits != 4:\n        raise ValueError(\"Currently, only 4bits is supported on CPU with AMX.\")\n    if quant_config.checkpoint_format == \"gptq_v2\":\n        raise ValueError(\"Currently, gptq_v2 is not supported on CPU with AMX.\")\n\n\nclass GPTQIntelAMXLinearScheme(GPTQLinearScheme):\n    \"\"\"Linear scheme for GPTQ on Intel CPU with AMX.\"\"\"\n\n    def _init_kernel(self, quant_config: GPTQConfig):\n        return GPTQIntelAMXLinearKernel(quant_config)\n\n    def create_weights(\n        self,\n        layer: torch.nn.Module,\n        input_size_per_partition: int,\n        output_partition_sizes: list[int],\n        input_size: int,\n        params_dtype: torch.dtype,\n        weight_loader,\n        **kwargs,\n    ):","sourceCodeStart":25,"sourceCodeEnd":61,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/gptq/schemes/gptq_cpu.py#L25-L61","documentation":"The Intel AMX GPTQ scheme does not support the gptq_v2 checkpoint format; _check_cpu_amx_support rejects quant_config.checkpoint_format == \"gptq_v2\" because the AMX kernel only understands the original (gptq) layout of qzeros/scales.","triggerScenarios":"Loading a GPTQ checkpoint whose quantization_config contains checkpoint_format: \"gptq_v2\" (common for auto-gptq >= 0.5 / newer GPTQModel exports) with the CPU AMX path; raised during create_weights.","commonSituations":"Newly exported GPTQ models default to gptq_v2; running them on Intel CPU with AMX support hits this immediately at model load.","solutions":["Re-export or convert the checkpoint with checkpoint_format: \"gptq\" (original format)","Use a GPU path where gptq_v2 is supported","Wait for/use a build with gptq_v2 AMX support, or requantize from the unquantized model with the legacy format"],"exampleFix":"# quantization_config.json before\n{\"quant_method\": \"gptq\", \"checkpoint_format\": \"gptq_v2\", ...}\n# after\n{\"quant_method\": \"gptq\", \"checkpoint_format\": \"gptq\", ...}","handlingStrategy":"validation","validationCode":"cfg = json.load(open(f\"{model_path}/quantization_config.json\"))\nif cfg.get(\"quant_method\") == \"gptq\" and cfg.get(\"checkpoint_format\") == \"gptq_v2\":\n    raise SystemExit(\"Re-export checkpoint with checkpoint_format='gptq' for CPU AMX\")","typeGuard":"def is_gptq_v2(cfg: dict) -> bool:\n    return cfg.get(\"checkpoint_format\", \"gptq\") == \"gptq_v2\"","tryCatchPattern":"try:\n    engine = Engine(model_path=model_path)\nexcept ValueError as e:\n    if \"gptq_v2\" in str(e):\n        sys.exit(\"Re-export the model with checkpoint_format='gptq' or run on GPU\")\n    raise","preventionTips":["Check quantization_config.json's checkpoint_format before selecting the CPU AMX backend","Keep legacy-format GPTQ exports for CPU/AMX fleets"],"tags":["gptq","checkpoint-format","gptq-v2","cpu","amx"],"backgroundTag":"unsupported-quantization-config","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}