sgl-project/sglang · error · ValueError

SGLang's AutoRound CPU inference path currently supports onl

Error message

SGLang's AutoRound CPU inference path currently supports only 4-bit AWQ/GPTQ checkpoints because it uses the Intel AMX INT4 backend, but got {weight_bits}-bit.

What it means

check_cpu_support gates SGLang's AutoRound CPU path to 4-bit checkpoints because it is implemented on the Intel AMX INT4 backend. Any other bit-width (e.g. 8-bit) on CPU raises before loading.

Source

Thrown at python/sglang/srt/layers/quantization/auto_round.py:283

                        layer_name.replace(fusion_key, sub_key) for sub_key in sub_keys
                    ]
                    sub_configs = [get_config(name, quantized) for name in sub_names]
                    if len(set(sub_configs)) == 1:
                        return sub_configs[0]
                    raise ValueError(
                        f"Fused module '{layer_name}' requires "
                        f"consistent quant config for {sub_names}"
                    )

        # 5. Fallback or try a regular expression match
        return get_config(layer_name, quantized)

    def check_quantized(self, weight_bits: int) -> bool:
        return weight_bits < 16

    def check_cpu_support(self, weight_bits: int) -> None:
        if weight_bits != 4:
            raise ValueError(
                "SGLang's AutoRound CPU inference path currently supports "
                "only 4-bit AWQ/GPTQ checkpoints because it uses the Intel "
                f"AMX INT4 backend, but got {weight_bits}-bit."
            )
        if not _is_cpu_amx_available:
            raise ValueError(_CPU_AMX_REQUIRED_MSG)

    def log_gptq_default_assumptions_once(self) -> None:
        if self._logged_gptq_default_assumptions or not self.gptq_defaulted_config_keys:
            return
        self._logged_gptq_default_assumptions = True
        default_summary = {
            key: _GPTQ_DEFAULTS[key] for key in self.gptq_defaulted_config_keys
        }
        logger.info(
            "AutoRound GPTQ config does not specify %s; using SGLang defaults %s.",
            ", ".join(self.gptq_defaulted_config_keys),
            default_summary,

View on GitHub (pinned to 0132848349)

Solutions

Use a 4-bit AutoRound checkpoint for CPU inference
Serve the 8-bit model on GPU, or use a CPU-supported w8a8 path if available for the quant method
Re-export from AutoRound with 4-bit weights

Example fix

# before
--quantization auto_round   # 8-bit ckpt on CPU
# after
--quantization auto_round   # with 4-bit re-exported ckpt

Defensive patterns

Strategy: type-guard

Validate before calling

from sglang.srt.utils import is_cpu
if is_cpu():
    assert quant_cfg["weight_bits"] == 4, "CPU AutoRound requires 4-bit"

Type guard

def cpu_autoround_ok(weight_bits: int, is_cpu: bool) -> bool:
    return (not is_cpu) or weight_bits == 4

Try / catch

try:
    method.create_weights(...)  # or launch server
except ValueError as e:
    if "AMX INT4" in str(e): switch_to_4bit_checkpoint_or_gpu()
    raise

Prevention

Maintain separate 4-bit checkpoints for CPU AMX serving
Gate model selection on detected hardware in deployment scripts

When it happens

Trigger: Serving an 8-bit (or other) AutoRound AWQ/GPTQ checkpoint with --quantization auto_round on a CPU/AMX machine; apply_awq_quant_layer/apply_gptq_quant_layer call check_cpu_support during weight loading.

Common situations: Reusing a GPU-tuned 8-bit AutoRound model on CPU servers; Intel Xeon AMX deployments assuming W8A8 works for AutoRound.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/c90454e06282f12f. Report an issue: GitHub.