sgl-project/sglang · critical · ValueError

Serialized W4A4 checkpoints require CUDA compute capability

Error message

Serialized W4A4 checkpoints require CUDA compute capability >= {self.get_min_capability() / 10:.1f}; got {capability.to_int() / 10:.1f}

What it means

KitchenW4A4Config.__init__ enforces a minimum CUDA compute capability (get_min_capability()). W4A4 kernels need newer GPU instructions (typically sm_80+/sm_100 depending on the build), so older cards are rejected at load.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/quantization/configs/kitchen_w4a4_config.py:44

_QUANT_GROUP_SIZE = 64
_SUPPORTED_CONVROT_GROUP_SIZES = (16, 64, 256)
_SUPPORTED_LINEAR_DTYPES = ("int4", "int8")


class KitchenW4A4Config(QuantizationConfig):
    """Dispatch serialized W4A4 linears and their optional INT8 companions."""

    def __init__(self, layer_markers: dict[str, dict[str, Any]]) -> None:
        super().__init__()
        if current_platform.is_mps():
            raise ValueError("Serialized W4A4 checkpoints are not supported on MPS")
        if current_platform.is_cuda():
            capability = current_platform.get_device_capability()
            if (
                capability is not None
                and capability.to_int() < self.get_min_capability()
            ):
                raise ValueError(
                    "Serialized W4A4 checkpoints require CUDA compute capability "
                    f">= {self.get_min_capability() / 10:.1f}; got "
                    f"{capability.to_int() / 10:.1f}"
                )
        self.layer_markers = layer_markers
        self.checkpoint_uses_native_qkv_layout = True
        self.selected: list[str] = []
        int8_markers = {
            prefix: marker
            for prefix, marker in layer_markers.items()
            if marker.get("format") == "int8_tensorwise"
        }
        self._int8_config = (
            KitchenInt8Config(layer_markers=int8_markers) if int8_markers else None
        )

        for prefix, marker in layer_markers.items():
            marker_format = marker.get("format")

View on GitHub (pinned to 0132848349)

Solutions

Run on a GPU meeting the minimum capability (check the class's get_min_capability)
Use a kitchen_int8 or unquantized export of the model

Example fix

// before
--device cuda  # on T4 (7.5)
// after
--device cuda  # on H100 (9.0), or load int8 checkpoint

Defensive patterns

Strategy: type-guard

Validate before calling

if torch.cuda.is_available():
    cap = torch.cuda.get_device_capability()
    assert cap[0] * 10 + cap[1] >= KitchenW4A4Config.get_min_capability()

Type guard

def gpu_meets_w4a4() -> bool:
    import torch
    if not torch.cuda.is_available():
        return False
    major, minor = torch.cuda.get_device_capability()
    return (major * 10 + minor) >= KitchenW4A4Config.get_min_capability()

Prevention

Check torch.cuda.get_device_capability() before selecting W4A4 checkpoints

When it happens

Trigger: Loading a W4A4 checkpoint on a CUDA GPU whose capability.to_int() is below get_min_capability(), e.g. a Turing (7.5) or Pascal card.

Common situations: Running on older datacenter GPUs (T4, V100) or older consumer cards; Docker images with older torch that misreport capability.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/ed2d6621336f6d56. Report an issue: GitHub.