hankcs/HanLP · error · ValueError

embed_dim must be divisible by num_heads (got `embed_dim`: {

Error message

embed_dim must be divisible by num_heads (got `embed_dim`: {self.embed_dim} and `num_heads`: {num_heads}).

What it means

ScalarMix mixes exactly mixture_size tensors with learned scalar weights; if you explicitly pass initial_scalar_parameters its length must equal mixture_size. A mismatch means the init weights don't map 1:1 to the tensors being mixed and the constructor raises ValueError.

Source

Thrown at hanlp/components/amr/amrbart/model_interface/modeling_bart.py:158

class BartAttention(nn.Module):
    """Multi-headed attention from 'Attention Is All You Need' paper"""

    def __init__(
        self,
        embed_dim: int,
        num_heads: int,
        dropout: float = 0.0,
        is_decoder: bool = False,
        bias: bool = True,
    ):
        super().__init__()
        self.embed_dim = embed_dim
        self.num_heads = num_heads
        self.dropout = dropout
        self.head_dim = embed_dim // num_heads

        if (self.head_dim * num_heads) != self.embed_dim:
            raise ValueError(
                f"embed_dim must be divisible by num_heads (got `embed_dim`: {self.embed_dim}"
                f" and `num_heads`: {num_heads})."
            )
        self.scaling = self.head_dim**-0.5
        self.is_decoder = is_decoder

        self.k_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
        self.v_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
        self.q_proj = nn.Linear(embed_dim, embed_dim, bias=bias)
        self.out_proj = nn.Linear(embed_dim, embed_dim, bias=bias)

    def _shape(self, tensor: torch.Tensor, seq_len: int, bsz: int):
        return tensor.view(bsz, seq_len, self.num_heads, self.head_dim).transpose(1, 2).contiguous()

    def forward(
        self,
        hidden_states: torch.Tensor,
        key_value_states: Optional[torch.Tensor] = None,

View on GitHub (pinned to ddb1299bdd)

Solutions

1. Omit initial_scalar_parameters (defaults to 0.0 for each mixture element)
2. Set len(initial_scalar_parameters) == mixture_size
3. Generate the list programmatically: [0.0] * n_layers

Example fix

# before
ScalarMix(mixture_size=4, initial_scalar_parameters=[0.0, 0.0])
# after
ScalarMix(mixture_size=4, initial_scalar_parameters=[0.0] * 4)

if initial_scalar_parameters is not None:
    assert len(initial_scalar_parameters) == mixture_size, (len(initial_scalar_parameters), mixture_size)

try:
    mix = ScalarMix(mixture_size, initial_scalar_parameters=params)
except ValueError:
    mix = ScalarMix(mixture_size)  # default 0.0 init

When it happens

Trigger: Constructing ScalarMix(mixture_size=2, initial_scalar_parameters=[0.0, 1.0, 2.0]) or any combination where the parameter list length differs from mixture_size.

Common situations: Hand-writing ELMo/transformer-layer-mixing configs; increasing the number of layers mixed (e.g. adding a layer to average) without updating the initial scalar parameters; copying configs between models with different layer counts.

Understand the failure class

Background: Config validation failed: what "invalid value for {key}" and settings-rejection errors mean across 19 open-source libraries — this error's family across 19 libraries.

Related errors

• Unsupported argument type: {item}
• Unrecognized mapper type {mapper}
• self.model.config.pad_token_id has to be defined.
• Attention weights should be of size {(bsz * self.num_heads,
• You cannot specify both input_ids and inputs_embeds at the s

AI-assisted analysis of hankcs/HanLP@ddb1299bdd (2026-08-27). Data as JSON: /api/errors/5cc3efbf970c74f6. Report an issue: GitHub.