sgl-project/sglang · error · ValueError

Currently, gptq_v2 is not supported on CPU with AMX.

Error message

Currently, gptq_v2 is not supported on CPU with AMX.

What it means

The Intel AMX GPTQ scheme does not support the gptq_v2 checkpoint format; _check_cpu_amx_support rejects quant_config.checkpoint_format == "gptq_v2" because the AMX kernel only understands the original (gptq) layout of qzeros/scales.

Source

Thrown at python/sglang/srt/layers/quantization/gptq/schemes/gptq_cpu.py:43

if TYPE_CHECKING:
    from sglang.srt.layers.moe.token_dispatcher import StandardDispatchOutput
    from sglang.srt.layers.quantization.gptq.gptq import GPTQConfig

__all__ = ["GPTQIntelAMXLinearScheme", "GPTQIntelAMXMoEScheme"]


def _check_cpu_amx_support(quant_config: GPTQConfig) -> None:
    if quant_config.desc_act and not (
        quant_config.true_sequential and quant_config.static_groups
    ):
        raise ValueError(
            "Currently, desc_act (True) is only supported with sequential "
            "and static group on CPU with AMX."
        )
    if quant_config.weight_bits != 4:
        raise ValueError("Currently, only 4bits is supported on CPU with AMX.")
    if quant_config.checkpoint_format == "gptq_v2":
        raise ValueError("Currently, gptq_v2 is not supported on CPU with AMX.")


class GPTQIntelAMXLinearScheme(GPTQLinearScheme):
    """Linear scheme for GPTQ on Intel CPU with AMX."""

    def _init_kernel(self, quant_config: GPTQConfig):
        return GPTQIntelAMXLinearKernel(quant_config)

    def create_weights(
        self,
        layer: torch.nn.Module,
        input_size_per_partition: int,
        output_partition_sizes: list[int],
        input_size: int,
        params_dtype: torch.dtype,
        weight_loader,
        **kwargs,
    ):

View on GitHub (pinned to 0132848349)

Solutions

1. Re-export or convert the checkpoint with checkpoint_format: "gptq" (original format)
2. Use a GPU path where gptq_v2 is supported
3. Wait for/use a build with gptq_v2 AMX support, or requantize from the unquantized model with the legacy format

Example fix

# quantization_config.json before
{"quant_method": "gptq", "checkpoint_format": "gptq_v2", ...}
# after
{"quant_method": "gptq", "checkpoint_format": "gptq", ...}

cfg = json.load(open(f"{model_path}/quantization_config.json"))
if cfg.get("quant_method") == "gptq" and cfg.get("checkpoint_format") == "gptq_v2":
    raise SystemExit("Re-export checkpoint with checkpoint_format='gptq' for CPU AMX")

def is_gptq_v2(cfg: dict) -> bool:
    return cfg.get("checkpoint_format", "gptq") == "gptq_v2"

try:
    engine = Engine(model_path=model_path)
except ValueError as e:
    if "gptq_v2" in str(e):
        sys.exit("Re-export the model with checkpoint_format='gptq' or run on GPU")
    raise

When it happens

Trigger: Loading a GPTQ checkpoint whose quantization_config contains checkpoint_format: "gptq_v2" (common for auto-gptq >= 0.5 / newer GPTQModel exports) with the CPU AMX path; raised during create_weights.

Common situations: Newly exported GPTQ models default to gptq_v2; running them on Intel CPU with AMX support hits this immediately at model load.

Related errors

• Currently, only 4bits is supported on CPU with AMX.
• The input size is not aligned with the quantized weight shap
• The output size is not aligned with the quantized weight sha
• Invalid quantization method on CPU: {quantization}. Availabl
• SGLang's AutoRound CPU inference path currently supports onl

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/15908b1f63f00e5c. Report an issue: GitHub.