keras-team/keras · error · ValueError

`idf_weights` must be the same length as vocabulary. len(idf

Error message

`idf_weights` must be the same length as vocabulary. len(idf_weights) is {len(idf_weights)}; len(vocabulary) is {len(vocabulary)}

What it means

In TF-IDF mode every token needs exactly one idf weight; a length mismatch would misalign the scaling, so set_vocabulary enforces `len(idf_weights) == len(vocabulary)`.

Source

Thrown at keras/src/layers/preprocessing/index_lookup.py:560

                "OOV tokens, or include them only at the start of the "
                f"vocabulary in precisely this order: {special_tokens}. "
                f"Received: oov_token={self.oov_token} at "
                f"vocabulary index {oov_index}"
            )

        new_vocab_size = token_start + len(tokens)
        if self.max_tokens is not None and (new_vocab_size > self.max_tokens):
            raise ValueError(
                "Attempted to set a vocabulary larger than the maximum vocab "
                f"size. Received vocabulary size is {new_vocab_size}; "
                f"`max_tokens` is {self.max_tokens}."
            )
        self.lookup_table = self._lookup_table_from_tokens(tokens)
        self._record_vocabulary_size()

        if self.output_mode == "tf_idf" and idf_weights is not None:
            if len(vocabulary) != len(idf_weights):
                raise ValueError(
                    "`idf_weights` must be the same length as vocabulary. "
                    f"len(idf_weights) is {len(idf_weights)}; "
                    f"len(vocabulary) is {len(vocabulary)}"
                )
            idf_weights = self._convert_to_ndarray(idf_weights)
            if idf_weights.ndim != 1:
                raise ValueError(
                    "TF-IDF data must be a 1-index array. "
                    f"Received: type(idf_weights)={type(idf_weights)}"
                )

            # If the passed vocabulary has no special tokens, we need to pad the
            # front of idf_weights. We don't have real document frequencies for
            # these tokens so we will use an average of all idf_weights passed
            # in as a reasonable default.
            if found_special_tokens:
                front_padding = 0
                front_padding_value = 0

View on GitHub (pinned to 7a34a03db6)

Solutions

1. Recompute idf on the exact vocabulary being set — fit sklearn TfidfVectorizer on the same corpus and use its vocabulary and idf_ together.
2. Align explicitly before the call: `assert len(vocab) == len(idf_weights)`.

Example fix

# before
layer.set_vocabulary(new_vocab, idf_weights=old_idf)

# after
assert len(new_vocab) == len(idf), (len(new_vocab), len(idf))
layer.set_vocabulary(new_vocab, idf_weights=idf)

assert len(vocab) == len(idf_weights), (len(vocab), len(idf_weights))
layer.set_vocabulary(vocab, idf_weights=idf_weights)

When it happens

Trigger: `set_vocabulary(vocab, idf_weights=w)` where w was computed against a different version of the vocabulary (extra or missing tokens).

Common situations: Recomputing the vocabulary after idf weights were fitted; off-by-one from special-token slots present in one array but not the other.

Related errors

• `idf_weights` should only be set if `output_mode` is `'tf_id
• When specifying the `vocabulary` argument, in TF-IDF output
• `idf_weights` must be set if output_mode is 'tf_idf'.
• `idf_weights` should only be set if output_mode is `'tf_idf'
• output_mode `'tf_idf'` does not support loading a vocabulary

AI-assisted analysis of keras-team/keras@7a34a03db6 (2026-08-25). Data as JSON: /api/errors/fad8b7fd49f905a6. Report an issue: GitHub.