tensorflow/models · error · ValueError

The input size ({hidden_size}) is not a multiple of the numb

Error message

The input size ({hidden_size}) is not a multiple of the number of attention heads ({self._num_heads})

What it means

Error "The input size ({hidden_size}) is not a multiple of the number of attention heads ({self._num_heads})" thrown in tensorflow/models.

Source

Thrown at official/projects/longformer/longformer_encoder_block.py:131

    else:
      self._attention_initializer = self._kernel_initializer
    self._attention_axes = attention_axes

  def build(self, input_shape):
    if isinstance(input_shape, tf.TensorShape):
      input_tensor_shape = input_shape
    elif isinstance(input_shape, (list, tuple)):
      input_tensor_shape = tf.TensorShape(input_shape[0])
    else:
      raise ValueError(
          f"The type of input shape argument is not supported, got: "
          f"{type(input_shape)}")
    einsum_equation = "abc,cd->abd"
    if len(input_tensor_shape.as_list()) > 3:
      einsum_equation = "...bc,cd->...bd"
    hidden_size = input_tensor_shape[-1]
    if hidden_size % self._num_heads != 0:
      raise ValueError(
          f"The input size ({hidden_size}) is not a multiple of the number of attention "
          f"heads ({self._num_heads})")
    self._attention_head_size = int(hidden_size // self._num_heads)
    common_kwargs = dict(
        bias_initializer=self._bias_initializer,
        kernel_regularizer=self._kernel_regularizer,
        bias_regularizer=self._bias_regularizer,
        activity_regularizer=self._activity_regularizer,
        kernel_constraint=self._kernel_constraint,
        bias_constraint=self._bias_constraint)
    # TFLongformerSelfAttention + TFLongformerSelfOutput.dense
    self._attention_layer = LongformerAttention(
        # Longformer
        layer_id=self._layer_id,
        global_attention_size=self.global_attention_size,
        attention_window=self._attention_window,
        num_heads=self._num_heads,
        key_dim=self._attention_head_size,

View on GitHub (pinned to e006f5f0d5)

When it happens

Trigger: Thrown at official/projects/longformer/longformer_encoder_block.py:131 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of tensorflow/models@e006f5f0d5 (2026-08-24). Data as JSON: /api/errors/8fa04c2ef988b1e0. Report an issue: GitHub.