sgl-project/sglang · critical · ValueError

Unsupported quantized embedding marker for {prefix!r}: {mark

Error message

Unsupported quantized embedding marker for {prefix!r}: {marker}

What it means

KitchenW4A8Config.get_quant_method accepts a VocabParallelEmbedding only if its marker is format 'int8_tensorwise' AND _is_tensorwise_scalar is set. Other embedding markers (e.g. the rowwise variant used by comfy_nvfp4) are rejected.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/quantization/configs/kitchen_w4a8_config.py:99

    @classmethod
    def from_config(cls, config: dict[str, Any]) -> KitchenW4A8Config:
        raise ValueError(
            "kitchen_w4a8 is inferred from per-layer checkpoint metadata; "
            "it is not an online quantization method"
        )

    def get_quant_method(
        self, layer: torch.nn.Module, prefix: str
    ) -> QuantizeMethodBase | None:
        marker = self.layer_markers.get(prefix)
        if isinstance(layer, VocabParallelEmbedding):
            if marker is None:
                return None
            if marker.get("format") != "int8_tensorwise" or not marker.get(
                "_is_tensorwise_scalar"
            ):
                raise ValueError(
                    f"Unsupported quantized embedding marker for {prefix!r}: {marker}"
                )
            self.selected.append(prefix)
            return KitchenInt8EmbeddingMethod()
        if not isinstance(layer, LinearBase):
            return None
        if marker is None:
            return UnquantizedLinearMethod()
        if marker.get("format") != "asym_w4a8_int8":
            raise ValueError(f"Unsupported quantized linear marker for {prefix!r}")

        group_size = int(marker.get("group_size", 16))
        convrot_group_size = int(marker.get("convrot_groupsize", 256))
        if not self._supports_input_size(
            layer.input_size, group_size, convrot_group_size
        ):
            raise ValueError(
                f"Serialized W4A8 layer {prefix!r} has input size "

View on GitHub (pinned to 0132848349)

Solutions

1. Re-quantize embeddings as tensorwise-scalar int8 (format 'int8_tensorwise', _is_tensorwise_scalar: true)
2. Drop the embedding marker so it stays unquantized
3. Load under comfy_nvfp4 if the rowwise int8 embedding is intentional

Example fix

// before
{"format": "int8_tensorwise", "_is_rowwise": true}
// after
{"format": "int8_tensorwise", "_is_tensorwise_scalar": true}

m = layer_markers.get(prefix)
if isinstance(layer, VocabParallelEmbedding) and m is not None:
    assert m.get("format") == "int8_tensorwise" and m.get("_is_tensorwise_scalar"), m

def is_w4a8_embedding_marker(m: dict) -> bool:
    return m.get("format") == "int8_tensorwise" and bool(m.get("_is_tensorwise_scalar"))

When it happens

Trigger: Loading an embedding whose marker has _is_rowwise instead of _is_tensorwise_scalar, or a non-int8 format, under kitchen_w4a8.

Common situations: Reusing markers produced for a different quant config (comfy_nvfp4 rowwise embeddings) with the W4A8 loader.

Related errors

• Unsupported quantized embedding marker for {prefix!r}: {mark
• Unsupported Comfy W4A8 format for {prefix!r}: {marker_format
• Serialized W4A8 layer {prefix!r} must set convrot=true
• SGLang diffusion currently supports AutoRound auto_gptq chec
• AutoRound fused module {target!r} has inconsistent shard con

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/759ae49f3627bba7. Report an issue: GitHub.