sgl-project/sglang · error · ValueError

Expected hidden_size to be at least {self.variance_size_over

Error message

Expected hidden_size to be at least {self.variance_size_override}, but found: {hidden_size}

What it means

When a layernorm is created with variance_size_override, forward_native computes statistics over only the first variance_size_override elements of the last dimension. It requires hidden_size >= variance_size_override; a smaller input triggers this ValueError, indicating the layer expects a wider activation than provided.

Source

Thrown at python/sglang/srt/layers/layernorm.py:814

            if post_residual_addition is not None:
                x = x + post_residual_addition.to(torch.float32)
            if self.fp32_residual:
                residual = x.clone()
            else:
                residual = x.to(orig_dtype)

        hidden_size = x.shape[-1]
        if hidden_size != self.hidden_size:
            raise ValueError(
                "Expected hidden_size to be "
                f"{self.hidden_size}, but found: {hidden_size}"
            )

        if self.variance_size_override is None:
            x_var = x
        else:
            if hidden_size < self.variance_size_override:
                raise ValueError(
                    "Expected hidden_size to be at least "
                    f"{self.variance_size_override}, but found: {hidden_size}"
                )

            x_var = x[..., : self.variance_size_override]

        variance = x_var.pow(2).mean(dim=-1, keepdim=True)
        x = x * torch.rsqrt(variance + self.variance_epsilon)

        if self.cast_x_before_out_mul:
            x = self.weight * x.to(orig_dtype)
        else:
            x = (x * self.weight).to(orig_dtype)

        if residual is None:
            return x
        else:
            return x, residual

View on GitHub (pinned to 0132848349)

Solutions

1. Verify variance_size_override in the model/config matches the actual activation width, and fix the config
2. Pass an input whose last dimension is at least variance_size_override
3. If the override is wrong for this layer, construct the layer with the correct value (or None)

Example fix

# before
layer = LayerNorm(hidden_size=2048, variance_size_override=4096)

# after
layer = LayerNorm(hidden_size=4096, variance_size_override=2048)

ov = layer.variance_size_override or layer.hidden_size
assert x.shape[-1] >= ov, f"need width >= {ov}, got {x.shape[-1]}"
out, res = layer.forward_native(x, residual)

def satisfies_variance_override(x: torch.Tensor, layer) -> bool:
    ov = getattr(layer, "variance_size_override", None)
    return ov is None or x.shape[-1] >= ov

try:
    out, res = layer.forward_native(x, residual)
except ValueError as e:
    if "variance_size_override" in str(e):
        raise ValueError(f"bad config: hidden_size {x.shape[-1]} < override; check model config") from e
    raise

When it happens

Trigger: Constructing the layer with variance_size_override = N and calling forward_native with x.shape[-1] < N — e.g. a hybrid-attention (Mamba/GDN) model where the mamba projection width or config disagrees with the override.

Common situations: Misparsed variance_size_override from a custom model config; models with per-layer variance overrides (GDN) where the checkpoint width was changed; tests feeding narrow tensors.

Related errors

• Unexpected initial_state_source shape: {initial_state_source
• rmsnorm_hf: unsupported hidden_size={hidden_size} (must be a
• Expected hidden_size to be at least {self.variance_size_over
• The pointers must be multiple of 16 bytes.
• The last dimension ({input.shape[-1]}) x itemsize ({input.dt

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/ac0f8ceef6295d76. Report an issue: GitHub.