tensorflow/models · error · ValueError

Block sparse attention does not support talking heads. Pleas

Error message

Block sparse attention does not support talking heads. Please set enable_talking_heads to False.

What it means

Error "Block sparse attention does not support talking heads. Please set enable_talking_heads to False." thrown in tensorflow/models.

Source

Thrown at official/nlp/modeling/layers/transformer_encoder_block.py:332

        value_dim=self._value_dim,
        dropout=self._attention_dropout_rate,
        use_bias=self._use_bias,
        kernel_initializer=self._attention_initializer,
        bias_initializer=tf_utils.clone_initializer(self._bias_initializer),
        attention_axes=self._attention_axes,
        output_shape=self._output_last_dim,
        softmax_robust_masking=self._softmax_robust_masking,
        name="self_attention",
    )
    common_kwargs = dict(
        bias_regularizer=self._bias_regularizer,
        activity_regularizer=self._activity_regularizer,
        kernel_constraint=self._kernel_constraint,
        bias_constraint=self._bias_constraint,
    )
    if self._src_block_size is not None:
      if self._enable_talking_heads:
        raise ValueError(
            "Block sparse attention does not support talking heads. Please"
            " set enable_talking_heads to False."
        )
      attention_layer_kwargs.update(
          src_block_size=self._src_block_size,
          tgt_block_size=self._tgt_block_size,
          use_sigmoid_attn=self._use_sigmoid_attn,
          sigmoid_attn_bias=self._sigmoid_attn_bias,
          num_kv_heads=self._num_kv_heads,
          name="block_sparse_attention",
      )
      attention_fn = block_sparse_attention.MultiHeadAttention
    elif self._num_kv_heads is not None:
      attention_layer_kwargs.update(
          num_kv_heads=self._num_kv_heads,
          enable_gqa_optimization=self._enable_gqa_optimization,
          name="multi_query_attention",
      )

View on GitHub (pinned to e006f5f0d5)

When it happens

Trigger: Thrown at official/nlp/modeling/layers/transformer_encoder_block.py:332 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of tensorflow/models@e006f5f0d5 (2026-08-24). Data as JSON: /api/errors/b05bfcc2187c7383. Report an issue: GitHub.