keras-team/keras · error · ValueError

When specifying the `vocabulary` argument, in TF-IDF output

Error message

When specifying the `vocabulary` argument, in TF-IDF output mode, the `idf_weights` argument must also be provided.

What it means

When a vocabulary is passed directly in the constructor with output_mode='tf_idf', the layer builds its lookup table immediately, so the idf weights must be supplied at the same moment. A frozen TF-IDF vocabulary has no later adapt path.

Source

Thrown at keras/src/layers/preprocessing/index_lookup.py:292

            elif self.num_oov_indices == 1:
                # If there is only one OOV index, we can set that index as the
                # default value of the index_lookup table.
                self._default_value = self._oov_start_index()
            else:
                # If we have multiple OOV values, we need to do a further
                # hashing step; to make this easier, we set the OOV value to -1.
                # (This lets us do a vectorized add and cast to boolean to
                # determine locations where we need to do extra hashing.)
                self._default_value = -1
        if self.mask_token is not None:
            self._mask_key = tf.convert_to_tensor(mask_key, self._key_dtype)
            self._mask_value = tf.convert_to_tensor(
                mask_value, self._value_dtype
            )

        if self.output_mode == "tf_idf":
            if self._has_input_vocabulary and idf_weights is None:
                raise ValueError(
                    "When specifying the `vocabulary` argument, "
                    "in TF-IDF output mode, the `idf_weights` argument "
                    "must also be provided."
                )
            if idf_weights is not None:
                self.idf_weights = tf.Variable(
                    idf_weights,
                    dtype=backend.floatx(),
                    trainable=False,
                )
                self.idf_weights_const = self.idf_weights.value()

        if vocabulary is not None:
            self.set_vocabulary(vocabulary, idf_weights)
        else:
            # When restoring from a keras SavedModel, the loading code will
            # expect to find and restore a lookup_table attribute on the layer.
            # This table needs to be uninitialized as a StaticHashTable cannot

View on GitHub (pinned to 7a34a03db6)

Solutions

Pass `idf_weights` together with `vocabulary` in the constructor.
Or leave `vocabulary=None` and call `set_vocabulary(vocab, idf_weights=w)` afterwards.
When porting sklearn, use `vectorizer.idf_` for the weights.

Example fix

# before
layer = IndexLookup(vocabulary=vocab, output_mode='tf_idf')

# after
layer = IndexLookup(vocabulary=vocab, idf_weights=idf, output_mode='tf_idf')

Defensive patterns

Strategy: validation

Validate before calling

if output_mode == 'tf_idf' and vocabulary is not None:
    assert idf_weights is not None, 'tf_idf + vocabulary requires idf_weights'
layer = IndexLookup(vocabulary=vocabulary, idf_weights=idf_weights, output_mode=output_mode)

Prevention

When porting sklearn, always import vocabulary_ and idf_ as a pair.
Prefer set_vocabulary(vocab, idf_weights=...) over constructor vocabulary for tf_idf layers.

When it happens

Trigger: `IndexLookup(vocabulary=vocab, output_mode='tf_idf')` with no `idf_weights`.

Common situations: Porting a fitted sklearn TfidfVectorizer: importing `vocabulary_` but forgetting `idf_`; switching a working int-mode layer to tf_idf while keeping the same constructor call.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

AI-assisted analysis of keras-team/keras@7a34a03db6 (2026-08-25). Data as JSON: /api/errors/a3c6357a6b4a8ae8. Report an issue: GitHub.