huggingface/transformers · error · ValueError

invalid distribution {distribution}

Error message

invalid distribution {distribution}

What it means

Error "invalid distribution {distribution}" thrown in huggingface/transformers.

Source

Thrown at src/transformers/initialization.py:192

    fan_in, fan_out = torch.nn.init._calculate_fan_in_and_fan_out(tensor)
    if mode == "fan_in":
        denom = fan_in
    elif mode == "fan_out":
        denom = fan_out
    elif mode == "fan_avg":
        denom = (fan_in + fan_out) / 2

    variance = 1.0 / denom

    if distribution == "truncated_normal":
        trunc_normal_(tensor, std=math.sqrt(variance) / 0.87962566103423978)
    elif distribution == "normal":
        normal_(tensor, std=math.sqrt(variance))
    elif distribution == "uniform":
        bound = math.sqrt(3 * variance)
        uniform_(tensor, -bound, bound)
    else:
        raise ValueError(f"invalid distribution {distribution}")


def lecun_normal_(tensor):
    if not getattr(tensor, "_is_hf_initialized", False):
        _variance_scaling(tensor, mode="fan_in", distribution="truncated_normal")
    return tensor


def default_flax_embed_init_(tensor):
    if not getattr(tensor, "_is_hf_initialized", False):
        _variance_scaling(tensor, mode="fan_in", distribution="normal")
    return tensor


# Here, we need to check several modules imported, and hot patch all of them, as sometimes torch does
# something like `from torch.nn.init import xavier_uniform_` in their internals (e.g in torch.nn.modules.activations,
# where MultiHeadAttention lives), so the function name is binded at import time and just doing
# `setattr(torch.nn.init, name, globals()[name])` is thus not enough

View on GitHub (pinned to a597f97485)

Solutions

1. Use a supported distribution name (e.g. 'normal', 'uniform').
2. Check the initializer's allowed distribution list.

When it happens

Trigger: Raised in initialization utilities when an unsupported weight distribution name is requested.

Common situations: Typo or unknown value passed as the distribution argument of an init helper (expected normal/uniform/etc.).

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/84a03a7e076d9b73. Report an issue: GitHub.