sgl-project/sglang · critical · ValueError

Serialized W4A8 checkpoints require CUDA compute capability

Error message

Serialized W4A8 checkpoints require CUDA compute capability >= {self.get_min_capability() / 10:.1f}; got {capability.to_int() / 10:.1f}

What it means

KitchenW4A8Config.__init__ enforces get_min_capability() on CUDA; W4A8 dequant kernels require newer architectures and older GPUs are rejected at load with the actual vs required capability printed.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/quantization/configs/kitchen_w4a8_config.py:41

    VocabParallelEmbedding,
)
from sglang.multimodal_gen.runtime.platforms import current_platform


class KitchenW4A8Config(QuantizationConfig):
    """Dispatch each linear from its serialized ``asym_w4a8_int8`` marker."""

    def __init__(self, layer_markers: dict[str, dict[str, Any]]) -> None:
        super().__init__()
        if current_platform.is_mps():
            raise ValueError("Serialized W4A8 checkpoints are not supported on MPS")
        if current_platform.is_cuda():
            capability = current_platform.get_device_capability()
            if (
                capability is not None
                and capability.to_int() < self.get_min_capability()
            ):
                raise ValueError(
                    "Serialized W4A8 checkpoints require CUDA compute capability "
                    f">= {self.get_min_capability() / 10:.1f}; got "
                    f"{capability.to_int() / 10:.1f}"
                )
        self.layer_markers = layer_markers
        self.checkpoint_uses_native_qkv_layout = True
        self.selected: list[str] = []

        for prefix, marker in layer_markers.items():
            marker_format = marker.get("format")
            if marker_format == "int8_tensorwise" and marker.get(
                "_is_tensorwise_scalar"
            ):
                continue
            if marker_format != "asym_w4a8_int8":
                raise ValueError(
                    f"Unsupported Comfy W4A8 format for {prefix!r}: "
                    f"{marker_format!r}"

View on GitHub (pinned to 0132848349)

Solutions

Move inference to a GPU meeting the minimum capability
Fall back to a kitchen_int8 or fp16 export

Example fix

// before: T4 (7.5)
// after: A100/H100 (8.0/9.0) or int8 checkpoint

Defensive patterns

Strategy: type-guard

Validate before calling

if torch.cuda.is_available():
    major, minor = torch.cuda.get_device_capability()
    assert major * 10 + minor >= KitchenW4A8Config.get_min_capability()

Type guard

def gpu_meets_w4a8() -> bool:
    import torch
    if not torch.cuda.is_available():
        return False
    major, minor = torch.cuda.get_device_capability()
    return (major * 10 + minor) >= KitchenW4A8Config.get_min_capability()

Prevention

Print device capability at startup and gate checkpoint choice on it

When it happens

Trigger: Loading a W4A8 checkpoint on a CUDA GPU with capability below the minimum (e.g. sm_75 T4).

Common situations: Older GPU fleets; misconfigured containers hiding the real GPU model.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/0c439f8fcc7c268e. Report an issue: GitHub.