tensorflow/models · error · ValueError

In the case of `norm_first`, the residual connection should

Error message

In the case of `norm_first`, the residual connection should be done in the TransformerScaffold call function, not FFN's call function.

What it means

Error "In the case of `norm_first`, the residual connection should be done in the TransformerScaffold call function, not FFN's call function." thrown in tensorflow/models.

Source

Thrown at official/vision/modeling/layers/nn_blocks.py:2843

          source_attention_output)
    else:
      attention_output = self._attention_layer_norm(
          input_tensor +
          self._stochastic_depth(attention_output, training=training))

    if self._feedforward_block is None:
      intermediate_output = self._intermediate_dense(attention_output)
      intermediate_output = self._intermediate_activation_layer(
          intermediate_output)
      layer_output = self._output_dense(intermediate_output)
      layer_output = self._output_dropout(layer_output, training=training)
    else:
      layer_output = self._feedforward_block(
          attention_output, training=training)

    if self._norm_first:
      if self._ffn_has_residual_connection:
        raise ValueError(
            'In the case of `norm_first`, the residual connection should be'
            "done in the TransformerScaffold call function, not FFN's"
            'call function.')
      output = source_attention_output + self._stochastic_depth(  # pyrefly: ignore[unbound-name]
          layer_output, training=training)
    else:
      # During mixed precision training, layer norm output is always fp32 for
      # now. Casts fp32 for the subsequent add.
      layer_output = tf.cast(layer_output, tf.float32)
      if self._ffn_has_residual_connection:
        output = self._stochastic_depth(layer_output, training=training)
      else:
        output = self._output_layer_norm(
            attention_output +
            self._stochastic_depth(layer_output, training=training))

    if self._return_attention_scores:
      return output, attention_scores

View on GitHub (pinned to e006f5f0d5)

Solutions

1. When norm_first is enabled, remove the residual connection from the FFN's call and let TransformerScaffold handle it.
2. Set the FFN's inner residual/add options consistently with norm_first.

When it happens

Trigger: Thrown at official/vision/modeling/layers/nn_blocks.py:2843 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of tensorflow/models@e006f5f0d5 (2026-08-24). Data as JSON: /api/errors/ca2c5eb02d72675e. Report an issue: GitHub.