sgl-project/sglang · critical · ValueError

Serialized W4A8 checkpoints are not supported on MPS

Error message

Serialized W4A8 checkpoints are not supported on MPS

What it means

KitchenW4A8Config.__init__ raises immediately on MPS: the asym_w4a8_int8 kernels have no Metal backend. This mirrors the W4A4 restriction and fails before any layer construction.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/quantization/configs/kitchen_w4a8_config.py:34

    QuantizeMethodBase,
)
from sglang.multimodal_gen.runtime.layers.quantization.kitchen_w4a8 import (
    KitchenInt8EmbeddingMethod,
    KitchenW4A8LinearMethod,
)
from sglang.multimodal_gen.runtime.layers.vocab_parallel_embedding import (
    VocabParallelEmbedding,
)
from sglang.multimodal_gen.runtime.platforms import current_platform


class KitchenW4A8Config(QuantizationConfig):
    """Dispatch each linear from its serialized ``asym_w4a8_int8`` marker."""

    def __init__(self, layer_markers: dict[str, dict[str, Any]]) -> None:
        super().__init__()
        if current_platform.is_mps():
            raise ValueError("Serialized W4A8 checkpoints are not supported on MPS")
        if current_platform.is_cuda():
            capability = current_platform.get_device_capability()
            if (
                capability is not None
                and capability.to_int() < self.get_min_capability()
            ):
                raise ValueError(
                    "Serialized W4A8 checkpoints require CUDA compute capability "
                    f">= {self.get_min_capability() / 10:.1f}; got "
                    f"{capability.to_int() / 10:.1f}"
                )
        self.layer_markers = layer_markers
        self.checkpoint_uses_native_qkv_layout = True
        self.selected: list[str] = []

        for prefix, marker in layer_markers.items():
            marker_format = marker.get("format")
            if marker_format == "int8_tensorwise" and marker.get(

View on GitHub (pinned to 0132848349)

Solutions

Run on a CUDA GPU
Use an MPS-compatible export (int8 or unquantized)

Example fix

// before
device="mps"
// after
device="cuda"

Defensive patterns

Strategy: type-guard

Validate before calling

import torch
if device == "mps" and quant_method == "kitchen_w4a8":
    raise SystemExit("W4A8 needs CUDA; pick an int8 export on MPS")

Type guard

def w4a8_supported_here() -> bool:
    import torch
    return torch.cuda.is_available()

Prevention

Map quant method to required device backend in your model registry

When it happens

Trigger: Instantiating KitchenW4A8Config on Apple Silicon with torch device mps.

Common situations: Local development on M-series Macs with a W4A8-quantized Comfy checkpoint.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/417a12b33ff6f57e. Report an issue: GitHub.