sgl-project/sglang · error · ValueError

Expected hidden_size to be {self.hidden_size}, but found: {h

Error message

Expected hidden_size to be {self.hidden_size}, but found: {hidden_size}

What it means

forward_native on a layernorm module (with variance_size_override, e.g. GDN/partial-variance norms) validates that the last dimension of the input equals the hidden_size the layer was constructed with. A mismatch means the module was built for a different width than the activations being fed to it — typically a config/feature-mismatch bug rather than a runtime data issue.

Source

Thrown at python/sglang/srt/layers/layernorm.py:805

        post_residual_addition: Optional[torch.Tensor] = None,
        quant_linear: Optional[nn.Module] = None,
    ) -> Union[torch.Tensor, Tuple[torch.Tensor, torch.Tensor]]:
        if not x.is_contiguous():
            x = x.contiguous()
        orig_dtype = self.override_orig_dtype or x.dtype
        x = x.to(torch.float32)
        if residual is not None:
            x = x + residual.to(torch.float32)
            if post_residual_addition is not None:
                x = x + post_residual_addition.to(torch.float32)
            if self.fp32_residual:
                residual = x.clone()
            else:
                residual = x.to(orig_dtype)

        hidden_size = x.shape[-1]
        if hidden_size != self.hidden_size:
            raise ValueError(
                "Expected hidden_size to be "
                f"{self.hidden_size}, but found: {hidden_size}"
            )

        if self.variance_size_override is None:
            x_var = x
        else:
            if hidden_size < self.variance_size_override:
                raise ValueError(
                    "Expected hidden_size to be at least "
                    f"{self.variance_size_override}, but found: {hidden_size}"
                )

            x_var = x[..., : self.variance_size_override]

        variance = x_var.pow(2).mean(dim=-1, keepdim=True)
        x = x * torch.rsqrt(variance + self.variance_epsilon)

View on GitHub (pinned to 0132848349)

Solutions

Ensure x.shape[-1] matches the hidden_size passed at module construction (check model config)
In tests, build inputs as torch.randn(..., module.hidden_size) instead of hardcoded sizes
Audit recent changes to hidden_size/quant config that feed this layer

Example fix

# before
x = torch.randn(B, T, 8192, device="cuda")
out, _ = layer.forward_native(x, residual)

# after
x = torch.randn(B, T, layer.hidden_size, device="cuda", dtype=layer.norm_weight.dtype)
out, _ = layer.forward_native(x, residual)

Defensive patterns

Strategy: validation

Validate before calling

assert x.shape[-1] == layer.hidden_size, (
    f"input width {x.shape[-1]} != layer hidden_size {layer.hidden_size}")
out, res = layer.forward_native(x, residual)

Type guard

def matches_hidden_size(x: torch.Tensor, layer) -> bool:
    return x.ndim >= 1 and x.shape[-1] == layer.hidden_size

Try / catch

try:
    out, res = layer.forward_native(x, residual)
except ValueError as e:
    if "Expected hidden_size" in str(e):
        raise RuntimeError(f"width mismatch feeding {type(layer).__name__}: {e}") from e
    raise

Prevention

Derive test input sizes from module attributes, not literals
Validate x.shape[-1] at module entry in debug builds
Keep model config single-sourced to avoid hidden_size drift

When it happens

Trigger: Constructing the layernorm with one hidden_size (from model config) and calling forward_native on an x whose x.shape[-1] differs — e.g. wrong config, quantized projection changing the feature width, or manually calling forward with a test tensor of the wrong size.

Common situations: Model config mismatches after editing hidden_size; unit tests constructing random inputs without matching hidden_size; variants (e.g. vision towers vs text) sharing a module with different widths.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/6f64ee429a0ef00f. Report an issue: GitHub.