tensorflow/models · error · ValueError

Linformer and Sigmoid attention are not supported in ReZero

Error message

Linformer and Sigmoid attention are not supported in ReZero Transformer.

What it means

Error "Linformer and Sigmoid attention are not supported in ReZero Transformer." thrown in tensorflow/models.

Source

Thrown at official/nlp/modeling/layers/rezero_transformer.py:127

    self._dropout_rate = dropout_rate
    self._output_range = output_range
    self._kernel_initializer = tf_keras.initializers.get(kernel_initializer)
    self._bias_initializer = tf_keras.initializers.get(bias_initializer)
    self._kernel_regularizer = tf_keras.regularizers.get(kernel_regularizer)
    self._bias_regularizer = tf_keras.regularizers.get(bias_regularizer)
    self._kernel_constraint = tf_keras.constraints.get(kernel_constraint)
    self._bias_constraint = tf_keras.constraints.get(bias_constraint)
    self._use_layer_norm = use_layer_norm
    self._share_rezero = share_rezero
    self._num_kv_heads = num_kv_heads
    self._src_block_size = src_block_size
    self._tgt_block_size = tgt_block_size
    self._linformer_dim = linformer_dim
    self._linformer_shared_kv_projection = linformer_shared_kv_projection
    self._use_sigmoid_attn = use_sigmoid_attn
    self._sigmoid_attn_bias = sigmoid_attn_bias
    if self._linformer_dim is not None or self._use_sigmoid_attn:
      raise ValueError(
          "Linformer and Sigmoid attention are not supported in ReZero"
          " Transformer."
      )
    if self._num_kv_heads is not None and self._src_block_size is not None:
      raise ValueError(
          "Block sparse attention does not support Multi-query attention."
          " Specify only one of them."
      )

  def build(self, input_shape):
    if isinstance(input_shape, tf.TensorShape):
      input_tensor_shape = input_shape
    elif isinstance(input_shape, (list, tuple)):
      input_tensor_shape = tf.TensorShape(input_shape[0])
    else:
      raise ValueError(
          "The type of input shape argument is not supported, got: %s"
          % type(input_shape)

View on GitHub (pinned to e006f5f0d5)

When it happens

Trigger: Thrown at official/nlp/modeling/layers/rezero_transformer.py:127 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of tensorflow/models@e006f5f0d5 (2026-08-24). Data as JSON: /api/errors/765eb3836cfadf0a. Report an issue: GitHub.