huggingface/transformers · error · ValueError

The embed_dim ({self.embed_dim}) is not a multiple of the nu

Error message

The embed_dim ({self.embed_dim}) is not a multiple of the number of attention heads ({self.num_heads}).

What it means

ValueError from validate_architecture (part of @strict config validation) when the config defines head_dim, num_heads and embed_dim and head_dim * num_heads != embed_dim. The model would build a projection matrix of the wrong shape, so the mismatch is caught at config time; heterogeneous configs recurse into each per-layer config.

Source

Thrown at src/transformers/configuration_utils.py:503

        if self.output_attentions and self._attn_implementation not in ["eager", None]:
            raise ValueError(
                "The `output_attentions` attribute is not supported when using the `attn_implementation` set to "
                f"{self._attn_implementation}. Please set it to 'eager' instead."
            )

    def validate_architecture(self):
        """Part of `@strict`-powered validation. Validates the architecture of the config."""
        if self.is_heterogeneous:
            for config in self.per_layer_config:
                config.validate_architecture()
            return
        if (
            hasattr(self, "head_dim")
            and hasattr(self, "num_heads")
            and hasattr(self, "embed_dim")
            and self.head_dim * self.num_heads != self.embed_dim
        ):
            raise ValueError(
                f"The embed_dim ({self.embed_dim}) is not a multiple of the number of attention "
                f"heads ({self.num_heads})."
            )

    def validate_token_ids(self):
        """Part of `@strict`-powered validation. Validates the contents of the special tokens."""
        text_config = self.get_text_config(decoder=True)
        vocab_size = getattr(text_config, "vocab_size", None)
        if vocab_size is not None:
            # Check for all special tokens, e..g. pad_token_id, image_token_id, audio_token_id
            for name in text_config:
                value = getattr(text_config, name)
                if name.endswith("_token_id") and isinstance(value, int) and not 0 <= value < vocab_size:
                    # Can't be an exception until we can load configs that fail validation: several configs on the Hub
                    # store invalid special tokens, e.g. `pad_token_id=-1`
                    logger.warning_once(
                        f"Model config: {name} must be `None` or an integer within the vocabulary (between 0 "
                        f"and {vocab_size - 1}), got {value}. This may result in unexpected behavior."

View on GitHub (pinned to a597f97485)

Solutions

Make head_dim * num_heads equal embed_dim, e.g. set head_dim = embed_dim // num_heads
If you intended a different embed_dim, change embed_dim to head_dim * num_heads consistently
Validate the triple right after parsing any user-provided architecture spec

Example fix

# before
cfg = MyConfig(embed_dim=512, num_heads=8, head_dim=80)
# after
cfg = MyConfig(embed_dim=512, num_heads=8, head_dim=64)

Defensive patterns

Strategy: validation

Validate before calling

assert 'head_dim' not in params or params['head_dim'] * params['num_heads'] == params['embed_dim'], \
    'head_dim * num_heads must equal embed_dim'

Type guard

def dims_consistent(embed_dim: int, num_heads: int, head_dim: int) -> bool:
    return head_dim * num_heads == embed_dim

Try / catch

try:
    cfg.validate_architecture()
except ValueError as e:
    if 'not a multiple' in str(e):
        params['head_dim'] = params['embed_dim'] // params['num_heads']
        cfg = MyConfig(**params)
    else:
        raise

Prevention

Compute head_dim as embed_dim // num_heads instead of hardcoding it
Validate the dims triple whenever a config comes from search tooling or external YAML

When it happens

Trigger: Constructing a config with embed_dim=512, num_heads=8, head_dim=80 (80*8=640 != 512); editing one of the three fields (e.g. num_heads for a variant) without rebalancing the others; wrong values parsed from a foreign checkpoint.

Common situations: Architecture search scripts mutating num_heads or head_dim independently; converting weights from another framework that names dims differently; typos in config YAMLs.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/03c59d445c61a755. Report an issue: GitHub.