tensorflow/models · error · ValueError

hidden size %d cannot be smaller than embedding width %d.

Error message

hidden size %d cannot be smaller than embedding width %d.

What it means

Error "hidden size %d cannot be smaller than embedding width %d." thrown in tensorflow/models.

Source

Thrown at official/nlp/modeling/layers/mobile_bert_layers.py:503

        name='transform/dense')

    if hidden_size > embedding_width:
      if self._output_weights_use_proj:
        self.extra_output_weights = self.add_weight(
            'output_weights_proj',
            shape=(embedding_width, hidden_size),
            initializer=tf_utils.clone_initializer(self.initializer),
            trainable=True)
      else:
        self.extra_output_weights = self.add_weight(
            'extra_output_weights',
            shape=(self._vocab_size, hidden_size - embedding_width),
            initializer=tf_utils.clone_initializer(self.initializer),
            trainable=True)
    elif hidden_size == embedding_width:
      self.extra_output_weights = None
    else:
      raise ValueError(
          'hidden size %d cannot be smaller than embedding width %d.' %
          (hidden_size, embedding_width))

    self.layer_norm = tf_keras.layers.LayerNormalization(
        axis=-1, epsilon=1e-12, name='transform/LayerNorm')
    self.bias = self.add_weight(
        'output_bias/bias',
        shape=(self._vocab_size,),
        initializer='zeros',
        trainable=True)

    super(MobileBertMaskedLM, self).build(input_shape)

  def call(self, sequence_data, masked_positions):
    masked_lm_input = self._gather_indexes(sequence_data, masked_positions)
    lm_data = self.dense(masked_lm_input)
    lm_data = self.layer_norm(lm_data)
    if self.extra_output_weights is None:

View on GitHub (pinned to e006f5f0d5)

When it happens

Trigger: Thrown at official/nlp/modeling/layers/mobile_bert_layers.py:503 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of tensorflow/models@e006f5f0d5 (2026-08-24). Data as JSON: /api/errors/9a5e8f10bab20ddf. Report an issue: GitHub.