sgl-project/sglang · error · ValueError

Expected hidden_size to be at least {self.variance_size_over

Error message

Expected hidden_size to be at least {self.variance_size_override}, but found: {hidden_size}

What it means

When variance_size_override is set (variance computed over only the first N features), forward_native requires hidden_size >= variance_size_override so the slice x[..., :override] is valid.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/layernorm.py:171

            x = x.contiguous()
        orig_dtype = x.dtype
        x = x.to(torch.float32)
        if residual is not None:
            x = x + residual.to(torch.float32)
            residual = x.to(orig_dtype)

        hidden_size = x.shape[-1]
        if hidden_size != self.hidden_size:
            raise ValueError(
                "Expected hidden_size to be "
                f"{self.hidden_size}, but found: {hidden_size}"
            )

        if self.variance_size_override is None:
            x_var = x
        else:
            if hidden_size < self.variance_size_override:
                raise ValueError(
                    "Expected hidden_size to be at least "
                    f"{self.variance_size_override}, but found: {hidden_size}"
                )

            x_var = x[..., : self.variance_size_override]

        if x.device.type == "mps" and self.variance_size_override is None:
            weight = self.weight.to(dtype=torch.float32)
            x = F.rms_norm(
                x,
                (self.hidden_size,),
                weight,
                self.variance_epsilon,
            ).to(orig_dtype)
            if residual is None:
                return x
            return x, residual

View on GitHub (pinned to 0132848349)

Solutions

1. Set variance_size_override <= hidden_size (or None to use full width)
2. Update the override when hidden_size changes
3. Validate at model build time: assert variance_size_override is None or <= hidden_size

Example fix

# before
norm = RMSNorm(hidden_size=1024, variance_size_override=1280)
# after
norm = RMSNorm(hidden_size=1024, variance_size_override=1024)  # or None

if norm.variance_size_override is not None:
    assert x.shape[-1] >= norm.variance_size_override

When it happens

Trigger: Constructing RMSNorm with variance_size_override=N and calling forward with hidden_size < N; typically a mis-specified partial-variance norm config.

Common situations: Configs where the variance window was sized for a larger hidden size than the layer's actual width; after hidden_size shrink (distillation/pruning) without updating the override.

Related errors

• Expected hidden_size to be {self.hidden_size}, but found: {h
• Expected hidden_size to be {self.hidden_size}, but found: {h
• Expected hidden_size to be at least {self.variance_size_over
• {tensor_name}{context_clause} with shape {tensor.shape} cann
• Block sparse tensors{context} must have shapes (B, H, M) and

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/9164d2fdc9926db0. Report an issue: GitHub.