tensorflow/models · error · ValueError

Block sparse attention does not support Multi-query attentio

Error message

Block sparse attention does not support Multi-query attention. Specify only one of them.

What it means

Error "Block sparse attention does not support Multi-query attention. Specify only one of them." thrown in tensorflow/models.

Source

Thrown at official/nlp/modeling/layers/rezero_transformer.py:132

    self._bias_regularizer = tf_keras.regularizers.get(bias_regularizer)
    self._kernel_constraint = tf_keras.constraints.get(kernel_constraint)
    self._bias_constraint = tf_keras.constraints.get(bias_constraint)
    self._use_layer_norm = use_layer_norm
    self._share_rezero = share_rezero
    self._num_kv_heads = num_kv_heads
    self._src_block_size = src_block_size
    self._tgt_block_size = tgt_block_size
    self._linformer_dim = linformer_dim
    self._linformer_shared_kv_projection = linformer_shared_kv_projection
    self._use_sigmoid_attn = use_sigmoid_attn
    self._sigmoid_attn_bias = sigmoid_attn_bias
    if self._linformer_dim is not None or self._use_sigmoid_attn:
      raise ValueError(
          "Linformer and Sigmoid attention are not supported in ReZero"
          " Transformer."
      )
    if self._num_kv_heads is not None and self._src_block_size is not None:
      raise ValueError(
          "Block sparse attention does not support Multi-query attention."
          " Specify only one of them."
      )

  def build(self, input_shape):
    if isinstance(input_shape, tf.TensorShape):
      input_tensor_shape = input_shape
    elif isinstance(input_shape, (list, tuple)):
      input_tensor_shape = tf.TensorShape(input_shape[0])
    else:
      raise ValueError(
          "The type of input shape argument is not supported, got: %s"
          % type(input_shape)
      )

    if len(input_tensor_shape.as_list()) != 3:
      raise ValueError(
          "TransformerLayer expects a three-dimensional input of "

View on GitHub (pinned to e006f5f0d5)

When it happens

Trigger: Thrown at official/nlp/modeling/layers/rezero_transformer.py:132 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of tensorflow/models@e006f5f0d5 (2026-08-24). Data as JSON: /api/errors/da4f0e34f17ad4f1. Report an issue: GitHub.