tensorflow/models · error · ValueError

Hidden size ({}) must be divisible by the number of heads ({

Error message

Hidden size ({}) must be divisible by the number of heads ({}).

What it means

Error "Hidden size ({}) must be divisible by the number of heads ({})." thrown in tensorflow/models.

Source

Thrown at official/legacy/transformer/attention_layer.py:35

import tensorflow as tf, tf_keras

from official.modeling import tf_utils


class Attention(tf_keras.layers.Layer):
  """Multi-headed attention layer."""

  def __init__(self, hidden_size, num_heads, attention_dropout):
    """Initialize Attention.

    Args:
      hidden_size: int, output dim of hidden layer.
      num_heads: int, number of heads to repeat the same attention structure.
      attention_dropout: float, dropout rate inside attention for training.
    """
    if hidden_size % num_heads:
      raise ValueError(
          "Hidden size ({}) must be divisible by the number of heads ({})."
          .format(hidden_size, num_heads))

    super(Attention, self).__init__()
    self.hidden_size = hidden_size
    self.num_heads = num_heads
    self.attention_dropout = attention_dropout

  def build(self, input_shape):
    """Builds the layer."""
    # Layers for linearly projecting the queries, keys, and values.
    size_per_head = self.hidden_size // self.num_heads

    def _glorot_initializer(fan_in, fan_out):
      limit = math.sqrt(6.0 / (fan_in + fan_out))
      return tf_keras.initializers.RandomUniform(minval=-limit, maxval=limit)

    attention_initializer = _glorot_initializer(input_shape.as_list()[-1],

View on GitHub (pinned to e006f5f0d5)

When it happens

Trigger: Thrown at official/legacy/transformer/attention_layer.py:35 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of tensorflow/models@e006f5f0d5 (2026-08-24). Data as JSON: /api/errors/1066a7594e70fc14. Report an issue: GitHub.