sgl-project/sglang · error · NotImplementedError

Not support norm_type: {norm_type}

Error message

Not support norm_type: {norm_type}

What it means

_make_norm (kimi_k3_vl.py:410) builds the LayerNorm/RMSNorm used by Kimi-K3 vision encoder blocks and only supports norm_type values "layernorm" and "rmsnorm". Any other string raises NotImplementedError at encoder construction. It exists to fail fast on unsupported checkpoint normalization schemes.

Source

Thrown at python/sglang/srt/models/kimi_k3_vl.py:410

class MLP2(nn.Module):
    def __init__(self, dims: List[int], activation, bias: bool = True):
        super().__init__()
        assert len(dims) == 3
        self.fc0 = nn.Linear(dims[0], dims[1], bias=bias)
        self.fc1 = nn.Linear(dims[1], dims[2], bias=bias)
        self.activation = activation

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.fc1(self.activation(self.fc0(x)))


def _make_norm(norm_type: str, dim: int) -> nn.Module:
    if norm_type == "layernorm":
        return nn.LayerNorm(dim)
    if norm_type == "rmsnorm":
        return nn.RMSNorm(dim)
    raise NotImplementedError(f"Not support norm_type: {norm_type}")


class MoonViTEncoderLayer(nn.Module):
    def __init__(
        self,
        num_heads: int,
        hidden_dim: int,
        mlp_dim: int,
        qkv_hidden_size: Optional[int] = None,
        norm_type: str = "layernorm",
        *,
        activation=F.gelu,
        attn_bias: bool = False,
        linear_bias: bool = True,
        attention_backend: str = "sdpa",
        attention_workspace: Optional[torch.Tensor] = None,
    ):
        super().__init__()

View on GitHub (pinned to 0132848349)

Solutions

1. Inspect the checkpoint config's norm_type fields; align with an official Kimi-K3 revision
2. Extend _make_norm with the needed norm module if the checkpoint genuinely uses it
3. Re-download the checkpoint in case of corrupted/partial config files

Example fix

// before
raise NotImplementedError(f"Not support norm_type: {norm_type}")
// after (if extending)
if norm_type == "groupnorm":
    return nn.GroupNorm(min(32, dim), dim)

assert cfg["norm_type"] in {"layernorm", "rmsnorm"}, cfg["norm_type"]

def is_supported_norm(t: str) -> bool:
    return t in {"layernorm", "rmsnorm"}

When it happens

Trigger: Loading a Kimi-K3-VL checkpoint whose block config specifies a norm_type other than layernorm/rmsnorm (e.g. "dnorm", "groupnorm").

Common situations: Finetuned or experimental checkpoints that swap normalization; config drift between model revisions; hand-merged configs.

Related errors

• Not support pos_emb_type: {pos_emb_type}
• Not support merge_type: {self.merge_type}
• Not support activation_func: {activation_func}
• Norm type {self.norm_type} not implemented
• Kimi K3 uses its model-native structural tag implementation

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/60aa143ff990b0ab. Report an issue: GitHub.