tensorflow/models · error · ValueError

TransformerLayer expects a three-dimensional input of shape

Error message

TransformerLayer expects a three-dimensional input of shape [batch, sequence, width].

What it means

Error "TransformerLayer expects a three-dimensional input of shape [batch, sequence, width]." thrown in tensorflow/models.

Source

Thrown at official/nlp/modeling/layers/rezero_transformer.py:149

    if self._num_kv_heads is not None and self._src_block_size is not None:
      raise ValueError(
          "Block sparse attention does not support Multi-query attention."
          " Specify only one of them."
      )

  def build(self, input_shape):
    if isinstance(input_shape, tf.TensorShape):
      input_tensor_shape = input_shape
    elif isinstance(input_shape, (list, tuple)):
      input_tensor_shape = tf.TensorShape(input_shape[0])
    else:
      raise ValueError(
          "The type of input shape argument is not supported, got: %s"
          % type(input_shape)
      )

    if len(input_tensor_shape.as_list()) != 3:
      raise ValueError(
          "TransformerLayer expects a three-dimensional input of "
          "shape [batch, sequence, width]."
      )
    batch_size, sequence_length, hidden_size = input_tensor_shape

    if len(input_shape) == 2:
      mask_tensor_shape = tf.TensorShape(input_shape[1])
      expected_mask_tensor_shape = tf.TensorShape(
          [batch_size, sequence_length, sequence_length]
      )
      if not expected_mask_tensor_shape.is_compatible_with(mask_tensor_shape):
        raise ValueError(
            "When passing a mask tensor to TransformerLayer, the "
            "mask tensor must be of shape [batch, "
            "sequence_length, sequence_length] (here %s). Got a "
            "mask tensor of shape %s."
            % (expected_mask_tensor_shape, mask_tensor_shape)
        )

View on GitHub (pinned to e006f5f0d5)

When it happens

Trigger: Thrown at official/nlp/modeling/layers/rezero_transformer.py:149 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of tensorflow/models@e006f5f0d5 (2026-08-24). Data as JSON: /api/errors/ff816406639db360. Report an issue: GitHub.