sgl-project/sglang · critical · ValueError

Serialized kitchen_int8 layer {prefix!r} has input size {lay

Error message

Serialized kitchen_int8 layer {prefix!r} has input size {layer.input_size}, which is not divisible by its ConvRot group size {marker_group_size}

What it means

KitchenInt8Config.get_quant_method checks that the linear layer's input_size is divisible by the marker's convrot_group_size; the ConvRot INT8 kernel processes weights in groups, so a non-divisible size has no valid tiling.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/quantization/configs/kitchen_int8_config.py:112

            ignored_layers=cls.get_from_keys_or(config, ["ignored_layers"], None),
        )

    def get_quant_method(
        self, layer: torch.nn.Module, prefix: str
    ) -> QuantizeMethodBase | None:
        from sglang.multimodal_gen.runtime.layers.linear import LinearBase
        from sglang.multimodal_gen.runtime.layers.quantization.kitchen_int8 import (
            KitchenInt8LinearMethod,
        )

        if not isinstance(layer, LinearBase):
            return None
        if self.layer_markers is not None:
            marker_group_size = self._serialized_group_sizes.get(prefix)
            if marker_group_size is None:
                return UnquantizedLinearMethod()
            if layer.input_size % marker_group_size:
                raise ValueError(
                    f"Serialized kitchen_int8 layer {prefix!r} has input size "
                    f"{layer.input_size}, which is not divisible by its "
                    f"ConvRot group size {marker_group_size}"
                )
            self.selected.append(prefix)
            return KitchenInt8LinearMethod(
                self,
                group_size=marker_group_size,
                is_checkpoint_serialized=True,
            )
        if is_layer_skipped(
            prefix, self.ignored_layers, fused_mapping=self.packed_modules_mapping
        ):
            self.skipped.append(prefix)
            return UnquantizedLinearMethod()
        # The rotation partitions the input dim into fixed-size groups, so a
        # layer whose input does not divide evenly simply stays in BF16 rather
        # than failing the whole model. H3's adaln projections (in=2688) are

View on GitHub (pinned to 0132848349)

Solutions

1. Use a smaller supported convrot_groupsize that divides input_size (e.g. 128 -> a divisor)
2. Re-quantize with padding or a group size matching the layer geometry
3. Leave the layer unserialized (remove its marker) so it uses UnquantizedLinearMethod

Example fix

// before
{"convrot": true, "convrot_groupsize": 256}  # input_size=1536? ok; input_size=1000? fails
// after
{"convrot": true, "convrot_groupsize": 100}  # divides 1000 (if supported)

gs = marker["convrot_groupsize"]
assert layer.input_size % gs == 0, f"{layer.input_size} not divisible by {gs}"

def input_size_ok(input_size: int, gs: int) -> bool:
    return input_size % gs == 0

When it happens

Trigger: Loading a serialized kitchen_int8 layer whose input_size % convrot_groupsize != 0, e.g. input_size=1000 with group size 128.

Common situations: Non-standard hidden sizes in custom architectures; markers copied from a different layer with a larger group size.

Related errors

• Shape mismatch: A K={k}, B K={b.shape[1] * 2}
• The input size is not aligned with the quantized weight shap
• Serialized kitchen_int8 layer {prefix!r} must set convrot=tr
• Serialized kitchen_int8 layer {prefix!r} must declare convro
• Serialized W4A4 layer {prefix!r} has input size {layer.input

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/2e6d53c74f0c8da1. Report an issue: GitHub.