sgl-project/sglang · error · ValueError

Serialized W4A8 layer {prefix!r} has input size {layer.input

Error message

Serialized W4A8 layer {prefix!r} has input size {layer.input_size}, incompatible with group_size={group_size} and convrot_groupsize={convrot_group_size}

What it means

Thrown when a serialized W4A8 layer's input_size is not divisible compatible with the group_size and convrot_group_size from its marker (checked via _supports_input_size). W4A8 packing requires the input dimension to align with the weight grouping and the conv-rotation group size, otherwise dequantization kernels would read out of bounds.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/quantization/configs/kitchen_w4a8_config.py:116

            ):
                raise ValueError(
                    f"Unsupported quantized embedding marker for {prefix!r}: {marker}"
                )
            self.selected.append(prefix)
            return KitchenInt8EmbeddingMethod()
        if not isinstance(layer, LinearBase):
            return None
        if marker is None:
            return UnquantizedLinearMethod()
        if marker.get("format") != "asym_w4a8_int8":
            raise ValueError(f"Unsupported quantized linear marker for {prefix!r}")

        group_size = int(marker.get("group_size", 16))
        convrot_group_size = int(marker.get("convrot_groupsize", 256))
        if not self._supports_input_size(
            layer.input_size, group_size, convrot_group_size
        ):
            raise ValueError(
                f"Serialized W4A8 layer {prefix!r} has input size "
                f"{layer.input_size}, incompatible with group_size={group_size} "
                f"and convrot_groupsize={convrot_group_size}"
            )
        self.selected.append(prefix)
        return KitchenW4A8LinearMethod(
            group_size=group_size,
            convrot_group_size=convrot_group_size,
            has_codebook=bool(marker.get("_has_codebook")),
            has_correction=bool(marker.get("_has_correction")),
        )

    @staticmethod
    def _supports_input_size(
        input_size: int, group_size: int, convrot_group_size: int
    ) -> bool:
        return (
            group_size >= 4

View on GitHub (pinned to 0132848349)

Solutions

1. Check layer.input_size % group_size == 0 and input_size % convrot_groupsize == 0 (per _supports_input_size logic)
2. Re-quantize the model with a convrot_groupsize that divides the input size (e.g. 128 or the input size itself)
3. Leave that layer unquantized (remove its marker) and let it fall back to UnquantizedLinearMethod

Example fix

# before
marker = {"format": "asym_w4a8_int8", "group_size": 16, "convrot_groupsize": 256}
# layer.input_size = 5504 -> not a multiple of 256 -> ValueError

# after
marker = {"format": "asym_w4a8_int8", "group_size": 16, "convrot_groupsize": 172}
# 5504 % 172 == 0 -> passes

def supports_input_size(input_size: int, group_size: int = 16, convrot: int = 256) -> bool:
    return input_size % group_size == 0 and input_size % convrot == 0

if not supports_input_size(layer.input_size, marker.get("group_size", 16), marker.get("convrot_groupsize", 256)):
    layer.marker = None  # fall back to unquantized

When it happens

Trigger: get_quant_method on a LinearBase whose layer.input_size is not a multiple of (or otherwise incompatible with) group_size (default 16) and convrot_groupsize (default 256), e.g. input_size=1000 with group_size=16 and convrot_groupsize=256.

Common situations: Loading a checkpoint quantized for a different architecture width; non-standard hidden sizes (e.g. 3584, 5504) that are not multiples of 256; a marker with a custom convrot_groupsize that doesn't divide the input size.

Related errors

• SGLang diffusion currently supports AutoRound auto_gptq chec
• AutoRound fused module {target!r} has inconsistent shard con
• Parameter {param_name} not found in the model.
• Unsupported quantized embedding marker for {prefix!r}: {mark
• Unsupported quantized linear marker for {prefix!r}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/3003fe7e57f59ea8. Report an issue: GitHub.