sgl-project/sglang · warning

mean is more than 2 std from [a, b] in nn.init.trunc_normal_

Error message

mean is more than 2 std from [a, b] in nn.init.trunc_normal_. The distribution of values may be incorrect.

What it means

Warning from a vendored copy of nn.init.trunc_normal_: it fires when the requested mean is more than 2 standard deviations outside the truncation interval [a, b], meaning the truncated-normal initialization will be badly skewed (most mass piled at the nearest bound).

Source

Thrown at python/sglang/srt/models/deepseek_janus_pro.py:136

# From PyTorch internals
def _ntuple(n):
    def parse(x):
        if isinstance(x, collections.abc.Iterable) and not isinstance(x, str):
            return tuple(x)
        return tuple(repeat(x, n))

    return parse


def _trunc_normal_(tensor, mean, std, a, b):
    # Cut & paste from PyTorch official master until it's in a few official releases - RW
    # Method based on https://people.sc.fsu.edu/~jburkardt/presentations/truncated_normal.pdf
    def norm_cdf(x):
        # Computes standard normal cumulative distribution function
        return (1.0 + math.erf(x / math.sqrt(2.0))) / 2.0

    if (mean < a - 2 * std) or (mean > b + 2 * std):
        logger.warn(
            "mean is more than 2 std from [a, b] in nn.init.trunc_normal_. "
            "The distribution of values may be incorrect.",
            stacklevel=2,
        )

    # Values are generated by using a truncated uniform distribution and
    # then using the inverse CDF for the normal distribution.
    # Get upper and lower cdf values
    l = norm_cdf((a - mean) / std)
    u = norm_cdf((b - mean) / std)

    # Uniformly fill tensor with values from [l, u], then translate to
    # [2l-1, 2u-1].
    tensor.uniform_(2 * l - 1, 2 * u - 1)

    # Use inverse cdf transform for normal distribution to get truncated
    # standard normal
    if tensor.dtype in [torch.float16, torch.bfloat16]:

View on GitHub (pinned to 0132848349)

Solutions

1. Fix the arguments: either move mean inside [a-2σ, b+2σ] or widen [a,b]/reduce std
2. If you intended mass near a bound, use a uniform or constant init instead
3. Add a unit test asserting init stats (mean/std of initialized tensor) match intent

Example fix

# before
trunc_normal_tf_(module.weight, std=0.02, a=0.1, b=0.9)
# after
trunc_normal_tf_(module.weight, std=0.02, a=-0.04, b=0.04)

assert a - 2*std <= mean <= b + 2*std, f"mean={mean} outside [{a},{b}] by >2std={std}"

def valid_trunc_normal(mean, std, a, b) -> bool:
    return (a - 2*std) <= mean <= (b + 2*std)

When it happens

Trigger: Calling trunc_normal_tf_ / _trunc_normal_ with parameters where mean < a - 2*std or mean > b + 2*std — e.g. trunc_normal_(tensor, mean=0.0, std=0.02, a=0.1) style calls in Janus-Pro vision/init code.

Common situations: Copying init code and editing bounds/mean inconsistently; initializing weights intended to be near-zero but with positive bounds; silent numerics degradation causing poor model quality.

Related errors

• `dt_bias` must have {HV * K} elements (got {dt_bias.numel()}
• Invalid packed Q size {q_dim}: must be divisible by K={K}. K
• Backward pass is not implemented yet and we do not have plan
• The batch size is expected to be 1 rather than {q.shape[0]}
• The number of initial states is expected to be equal to the

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/eeb703b75f612f3b. Report an issue: GitHub.