huggingface/pytorch-image-models · warning

mean is more than 2 std from [a, b] in nn.init.trunc_normal_

Error message

mean is more than 2 std from [a, b] in nn.init.trunc_normal_. The distribution of values may be incorrect.

What it means

timm's vendored _trunc_normal_ (used by trunc_normal_ and trunc_normal_tf_) warns when the requested mean lies more than 2 standard deviations outside the truncation bounds [a, b] — the rejection-sampling-based fill then produces a badly skewed distribution instead of the intended truncated normal.

Source

Thrown at timm/layers/weight_init.py:27

    """Check if targeting meta device (explicit arg or context manager)."""
    if device is not None:
        return str(device) == 'meta'
    # Check context manager (PyTorch 2.0+)
    if hasattr(torch, 'get_default_device'):
        default_device = torch.get_default_device()
        return default_device is not None and default_device.type == 'meta'
    return False


def _trunc_normal_(tensor, mean, std, a, b):
    # Cut & paste from PyTorch official master until it's in a few official releases - RW
    # Method based on https://people.sc.fsu.edu/~jburkardt/presentations/truncated_normal.pdf
    def norm_cdf(x):
        # Computes standard normal cumulative distribution function
        return (1. + math.erf(x / math.sqrt(2.))) / 2.

    if (mean < a - 2 * std) or (mean > b + 2 * std):
        warnings.warn("mean is more than 2 std from [a, b] in nn.init.trunc_normal_. "
                      "The distribution of values may be incorrect.",
                      stacklevel=2)

    # Values are generated by using a truncated uniform distribution and
    # then using the inverse CDF for the normal distribution.
    # Get upper and lower cdf values
    l = norm_cdf((a - mean) / std)
    u = norm_cdf((b - mean) / std)

    # Uniformly fill tensor with values from [l, u], then translate to
    # [2l-1, 2u-1].
    tensor.uniform_(2 * l - 1, 2 * u - 1)

    # Use inverse cdf transform for normal distribution to get truncated
    # standard normal
    tensor.erfinv_()

    # Transform to proper mean, std

View on GitHub (pinned to 9a5261e31b)

Solutions

1. Recheck a/b semantics: they are absolute value bounds; set a=mean-2*std*something sensible or use a=-2, b=2 style wide bounds
2. Center the mean inside [a, b] (within 2 std)
3. Simply use default trunc_normal_(w, std=.02) which uses a=-2, b=2

Example fix

# before
trunc_normal_(w, mean=1.0, std=0.02, a=-0.2, b=0.2)  # warns, skewed init
# after
trunc_normal_(w, mean=0.0, std=0.02, a=-0.2, b=0.2)

assert a <= mean <= b and (mean - 2*std) >= a - 1e-9 and (mean + 2*std) <= b + 1e-9, 'mean too far from [a, b]'

When it happens

Trigger: trunc_normal_(tensor, mean=1.0, std=0.02, a=-0.2, b=0.2) — mean is 2+ std beyond b; zero std with nonzero mean offset; truncation bounds from a constant like 0.02 while mean is 0.5.

Common situations: Custom initializers ported from papers with unusual parametrization; misreading b as a multiplier instead of an absolute bound. Resulting weights still initialize but statistics are wrong, hurting training.

Related errors

• Input image must have positive dimensions, got H={height}, W
• Invalid class map file, expected a dict ({class_map_path}).
• Dataset length is unknown, please pass `num_samples` explici
• Found 0 images in subfolders of {root}. Supported image exte
• Invalid or corrupt tar info cache file {cache_path}.

AI-assisted analysis of huggingface/pytorch-image-models@9a5261e31b (2026-08-27). Data as JSON: /api/errors/377a17e5c0640130. Report an issue: GitHub.