{"record":{"id":"eeb703b75f612f3b","repo":"sgl-project/sglang","slug":"mean-is-more-than-2-std-from-a-b-in-nn-init-tru","errorCode":null,"errorMessage":"mean is more than 2 std from [a, b] in nn.init.trunc_normal_. The distribution of values may be incorrect.","messagePattern":"mean is more than 2 std from \\[a, b\\] in nn\\.init\\.trunc_normal_\\. The distribution of values may be incorrect\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"python/sglang/srt/models/deepseek_janus_pro.py","lineNumber":136,"sourceCode":"# From PyTorch internals\ndef _ntuple(n):\n    def parse(x):\n        if isinstance(x, collections.abc.Iterable) and not isinstance(x, str):\n            return tuple(x)\n        return tuple(repeat(x, n))\n\n    return parse\n\n\ndef _trunc_normal_(tensor, mean, std, a, b):\n    # Cut & paste from PyTorch official master until it's in a few official releases - RW\n    # Method based on https://people.sc.fsu.edu/~jburkardt/presentations/truncated_normal.pdf\n    def norm_cdf(x):\n        # Computes standard normal cumulative distribution function\n        return (1.0 + math.erf(x / math.sqrt(2.0))) / 2.0\n\n    if (mean < a - 2 * std) or (mean > b + 2 * std):\n        logger.warn(\n            \"mean is more than 2 std from [a, b] in nn.init.trunc_normal_. \"\n            \"The distribution of values may be incorrect.\",\n            stacklevel=2,\n        )\n\n    # Values are generated by using a truncated uniform distribution and\n    # then using the inverse CDF for the normal distribution.\n    # Get upper and lower cdf values\n    l = norm_cdf((a - mean) / std)\n    u = norm_cdf((b - mean) / std)\n\n    # Uniformly fill tensor with values from [l, u], then translate to\n    # [2l-1, 2u-1].\n    tensor.uniform_(2 * l - 1, 2 * u - 1)\n\n    # Use inverse cdf transform for normal distribution to get truncated\n    # standard normal\n    if tensor.dtype in [torch.float16, torch.bfloat16]:","sourceCodeStart":118,"sourceCodeEnd":154,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/models/deepseek_janus_pro.py#L118-L154","documentation":"Warning from a vendored copy of nn.init.trunc_normal_: it fires when the requested mean is more than 2 standard deviations outside the truncation interval [a, b], meaning the truncated-normal initialization will be badly skewed (most mass piled at the nearest bound).","triggerScenarios":"Calling trunc_normal_tf_ / _trunc_normal_ with parameters where mean < a - 2*std or mean > b + 2*std — e.g. trunc_normal_(tensor, mean=0.0, std=0.02, a=0.1) style calls in Janus-Pro vision/init code.","commonSituations":"Copying init code and editing bounds/mean inconsistently; initializing weights intended to be near-zero but with positive bounds; silent numerics degradation causing poor model quality.","solutions":["Fix the arguments: either move mean inside [a-2σ, b+2σ] or widen [a,b]/reduce std","If you intended mass near a bound, use a uniform or constant init instead","Add a unit test asserting init stats (mean/std of initialized tensor) match intent"],"exampleFix":"# before\ntrunc_normal_tf_(module.weight, std=0.02, a=0.1, b=0.9)\n# after\ntrunc_normal_tf_(module.weight, std=0.02, a=-0.04, b=0.04)","handlingStrategy":"validation","validationCode":"assert a - 2*std <= mean <= b + 2*std, f\"mean={mean} outside [{a},{b}] by >2std={std}\"","typeGuard":"def valid_trunc_normal(mean, std, a, b) -> bool:\n    return (a - 2*std) <= mean <= (b + 2*std)","tryCatchPattern":null,"preventionTips":["Sanity-check init arguments against bounds","Unit-test initialized tensor statistics"],"tags":["pytorch","initialization","numerics","trunc-normal"],"backgroundTag":"invalid-initialization-parameters","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}