tensorflow/models · error · ValueError

Block sparse attention only supports Multi-query attention.P

Error message

Block sparse attention only supports Multi-query attention.Please set num_kv_heads to 1 to enable MQA with block sparse attention.

What it means

Error "Block sparse attention only supports Multi-query attention.Please set num_kv_heads to 1 to enable MQA with block sparse attention." thrown in tensorflow/models.

Source

Thrown at official/nlp/modeling/layers/transformer_encoder_block.py:265

    self._diff_q_kv_att_layer_norm = diff_q_kv_att_layer_norm
    self._return_attention_scores = return_attention_scores
    self._num_kv_heads = num_kv_heads
    self._src_block_size = src_block_size
    self._tgt_block_size = tgt_block_size
    self._use_sigmoid_attn = use_sigmoid_attn
    self._sigmoid_attn_bias = sigmoid_attn_bias
    self._linformer_dim = linformer_dim
    self._linformer_shared_kv_projection = linformer_shared_kv_projection
    self._lowrank_query_seq_proj_dim = lowrank_query_seq_proj_dim
    self._enable_talking_heads = enable_talking_heads
    self._enable_gqa_optimization = enable_gqa_optimization
    self._softmax_robust_masking = softmax_robust_masking
    if (
        self._src_block_size is not None
        and self._num_kv_heads is not None
        and self._num_kv_heads != 1
    ):
      raise ValueError(
          "Block sparse attention only supports Multi-query attention.Please"
          " set num_kv_heads to 1 to enable MQA with block sparse attention."
      )
    if attention_initializer:
      self._attention_initializer = tf_keras.initializers.get(
          attention_initializer
      )
    else:
      self._attention_initializer = tf_utils.clone_initializer(
          self._kernel_initializer
      )
    self._attention_axes = attention_axes

    if self._diff_q_kv_att_layer_norm and not self._norm_first:
      raise ValueError(
          "Setting `diff_q_and_kv_attention_layer_norm` to True"
          "when `norm_first` is False is invalid."
      )

View on GitHub (pinned to e006f5f0d5)

When it happens

Trigger: Thrown at official/nlp/modeling/layers/transformer_encoder_block.py:265 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of tensorflow/models@e006f5f0d5 (2026-08-24). Data as JSON: /api/errors/42e9e0a97b9a9e46. Report an issue: GitHub.