tensorflow/models · error · ValueError

The hidden size (%d) is not a multiple of the number of atte

Error message

The hidden size (%d) is not a multiple of the number of attention heads (%d)

What it means

Error "The hidden size (%d) is not a multiple of the number of attention heads (%d)" thrown in tensorflow/models.

Source

Thrown at official/nlp/modeling/layers/transformer.py:261

    if cross_attention_cls is not None:
      self._cross_attention_cls = cross_attention_cls
      if self.multi_channel_cross_attention:
        logging.warning(
            "%s will be used for cross attention", cross_attention_cls
        )
    elif self.multi_channel_cross_attention:
      self._cross_attention_cls = multi_channel_attention.MultiChannelAttention
    else:
      self._cross_attention_cls = attention.MultiHeadAttention

  def build(self, input_shape):
    target_tensor_shape = tf.TensorShape(input_shape[0])
    if len(target_tensor_shape.as_list()) != 3:
      raise ValueError("TransformerLayer expects a three-dimensional input of "
                       "shape [batch, sequence, width].")
    hidden_size = target_tensor_shape[2]
    if hidden_size % self.num_attention_heads != 0:
      raise ValueError(
          "The hidden size (%d) is not a multiple of the number of attention "
          "heads (%d)" % (hidden_size, self.num_attention_heads))
    self.attention_head_size = int(hidden_size) // self.num_attention_heads
    common_kwargs = dict(
        kernel_regularizer=self._kernel_regularizer,
        bias_regularizer=self._bias_regularizer,
        activity_regularizer=self._activity_regularizer,
        kernel_constraint=self._kernel_constraint,
        bias_constraint=self._bias_constraint)
    # Self attention.
    self.self_attention = self._self_attention_cls(
        num_heads=self.num_attention_heads,
        key_dim=self.attention_head_size,
        dropout=self.attention_dropout_rate,
        use_bias=self._use_bias,
        kernel_initializer=tf_utils.clone_initializer(
            self._attention_initializer),
        bias_initializer=tf_utils.clone_initializer(self._bias_initializer),

View on GitHub (pinned to e006f5f0d5)

When it happens

Trigger: Thrown at official/nlp/modeling/layers/transformer.py:261 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of tensorflow/models@e006f5f0d5 (2026-08-24). Data as JSON: /api/errors/b1ca69ab1362c6ae. Report an issue: GitHub.