{"record":{"id":"377a17e5c0640130","repo":"huggingface/pytorch-image-models","slug":"mean-is-more-than-2-std-from-a-b-in-nn-init-tru","errorCode":null,"errorMessage":"mean is more than 2 std from [a, b] in nn.init.trunc_normal_. The distribution of values may be incorrect.","messagePattern":"mean is more than 2 std from \\[a, b\\] in nn\\.init\\.trunc_normal_\\. The distribution of values may be incorrect\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"timm/layers/weight_init.py","lineNumber":27,"sourceCode":"    \"\"\"Check if targeting meta device (explicit arg or context manager).\"\"\"\n    if device is not None:\n        return str(device) == 'meta'\n    # Check context manager (PyTorch 2.0+)\n    if hasattr(torch, 'get_default_device'):\n        default_device = torch.get_default_device()\n        return default_device is not None and default_device.type == 'meta'\n    return False\n\n\ndef _trunc_normal_(tensor, mean, std, a, b):\n    # Cut & paste from PyTorch official master until it's in a few official releases - RW\n    # Method based on https://people.sc.fsu.edu/~jburkardt/presentations/truncated_normal.pdf\n    def norm_cdf(x):\n        # Computes standard normal cumulative distribution function\n        return (1. + math.erf(x / math.sqrt(2.))) / 2.\n\n    if (mean < a - 2 * std) or (mean > b + 2 * std):\n        warnings.warn(\"mean is more than 2 std from [a, b] in nn.init.trunc_normal_. \"\n                      \"The distribution of values may be incorrect.\",\n                      stacklevel=2)\n\n    # Values are generated by using a truncated uniform distribution and\n    # then using the inverse CDF for the normal distribution.\n    # Get upper and lower cdf values\n    l = norm_cdf((a - mean) / std)\n    u = norm_cdf((b - mean) / std)\n\n    # Uniformly fill tensor with values from [l, u], then translate to\n    # [2l-1, 2u-1].\n    tensor.uniform_(2 * l - 1, 2 * u - 1)\n\n    # Use inverse cdf transform for normal distribution to get truncated\n    # standard normal\n    tensor.erfinv_()\n\n    # Transform to proper mean, std","sourceCodeStart":9,"sourceCodeEnd":45,"githubUrl":"https://github.com/huggingface/pytorch-image-models/blob/9a5261e31b3b5128526eb2658333b4c0a54464ae/timm/layers/weight_init.py#L9-L45","documentation":"timm's vendored _trunc_normal_ (used by trunc_normal_ and trunc_normal_tf_) warns when the requested mean lies more than 2 standard deviations outside the truncation bounds [a, b] — the rejection-sampling-based fill then produces a badly skewed distribution instead of the intended truncated normal.","triggerScenarios":"trunc_normal_(tensor, mean=1.0, std=0.02, a=-0.2, b=0.2) — mean is 2+ std beyond b; zero std with nonzero mean offset; truncation bounds from a constant like 0.02 while mean is 0.5.","commonSituations":"Custom initializers ported from papers with unusual parametrization; misreading b as a multiplier instead of an absolute bound. Resulting weights still initialize but statistics are wrong, hurting training.","solutions":["Recheck a/b semantics: they are absolute value bounds; set a=mean-2*std*something sensible or use a=-2, b=2 style wide bounds","Center the mean inside [a, b] (within 2 std)","Simply use default trunc_normal_(w, std=.02) which uses a=-2, b=2"],"exampleFix":"# before\ntrunc_normal_(w, mean=1.0, std=0.02, a=-0.2, b=0.2)  # warns, skewed init\n# after\ntrunc_normal_(w, mean=0.0, std=0.02, a=-0.2, b=0.2)","handlingStrategy":"validation","validationCode":"assert a <= mean <= b and (mean - 2*std) >= a - 1e-9 and (mean + 2*std) <= b + 1e-9, 'mean too far from [a, b]'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Prefer defaults: trunc_normal_(w, std=.02) uses a=-2, b=2","Remember a/b are absolute bounds, not multipliers of std"],"tags":["weight-init","trunc-normal","statistics","timm"],"backgroundTag":"invalid-initialization-parameters","analyzedSha":"9a5261e31b3b5128526eb2658333b4c0a54464ae","analyzedAt":"2026-08-27T02:34:25.417Z","schemaVersion":2},"datasetVersion":"2026-08-27T03:17:27.898Z"}