sgl-project/sglang · error · ValueError

Currently, only 4bits is supported on CPU with AMX.

Error message

Currently, only 4bits is supported on CPU with AMX.

What it means

GPTQ CPU/AMX kernel (GPTQIntelAMXLinearScheme) only supports 4-bit weights; the check in _check_cpu_amx_support rejects any quant_config.weight_bits other than 4 (e.g. 2-bit, 3-bit, 8-bit GPTQ checkpoints) before weight creation.

Source

Thrown at python/sglang/srt/layers/quantization/gptq/schemes/gptq_cpu.py:41

from .gptq_scheme import GPTQMoESchemeBase

if TYPE_CHECKING:
    from sglang.srt.layers.moe.token_dispatcher import StandardDispatchOutput
    from sglang.srt.layers.quantization.gptq.gptq import GPTQConfig

__all__ = ["GPTQIntelAMXLinearScheme", "GPTQIntelAMXMoEScheme"]


def _check_cpu_amx_support(quant_config: GPTQConfig) -> None:
    if quant_config.desc_act and not (
        quant_config.true_sequential and quant_config.static_groups
    ):
        raise ValueError(
            "Currently, desc_act (True) is only supported with sequential "
            "and static group on CPU with AMX."
        )
    if quant_config.weight_bits != 4:
        raise ValueError("Currently, only 4bits is supported on CPU with AMX.")
    if quant_config.checkpoint_format == "gptq_v2":
        raise ValueError("Currently, gptq_v2 is not supported on CPU with AMX.")


class GPTQIntelAMXLinearScheme(GPTQLinearScheme):
    """Linear scheme for GPTQ on Intel CPU with AMX."""

    def _init_kernel(self, quant_config: GPTQConfig):
        return GPTQIntelAMXLinearKernel(quant_config)

    def create_weights(
        self,
        layer: torch.nn.Module,
        input_size_per_partition: int,
        output_partition_sizes: list[int],
        input_size: int,
        params_dtype: torch.dtype,
        weight_loader,

View on GitHub (pinned to 0132848349)

Solutions

1. Use a 4-bit GPTQ quantized checkpoint (wbits=4) for CPU AMX inference
2. Requantize the model to 4-bit GPTQ with static group size
3. Fall back to a different backend/device (e.g. GPU or a non-AMX CPU kernel) that supports the checkpoint's bit width

Example fix

# before
model = AutoModelForCausalLM.from_pretrained("org/model-gptq-8bit")  # weight_bits=8

# after
model = ...  # use "org/model-gptq-4bit" with quantization_config weight_bits=4

cfg = GPTQConfig.from_pretrained(model_path)
if cfg.quant_method == "gptq" and is_cpu_amx_backend() and cfg.weight_bits != 4:
    raise SystemExit("Need a 4-bit GPTQ checkpoint for CPU AMX; got " f"{cfg.weight_bits}-bit")

def is_amx_compatible_gptq(cfg: GPTQConfig) -> bool:
    return cfg.weight_bits == 4 and cfg.checkpoint_format != "gptq_v2" and not (cfg.desc_act and not cfg.static_groups)

When it happens

Trigger: Loading a model quantized with GPTQ wbits=2, 3, or 8 on an Intel CPU using the AMX path (e.g. quant_config with weight_bits != 4 while running with the Intel AMX scheme); raised from create_weights via _check_cpu_amx_support.

Common situations: Running a GPTQ-Int8 or GPTQ-2bit HuggingFace checkpoint on Xeon servers with AMX tiles enabled; switching a deployment from GPU to CPU without requantizing the model.

Related errors

• The input size is not aligned with the quantized weight shap
• Invalid quantization method on CPU: {quantization}. Availabl
• SGLang's AutoRound CPU inference path currently supports onl
• Currently, gptq_v2 is not supported on CPU with AMX.
• The output size is not aligned with the quantized weight sha

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/830667361156c5d6. Report an issue: GitHub.