keras-team/keras · error · ValueError

Attempted to set a vocabulary larger than the maximum vocab

Error message

Attempted to set a vocabulary larger than the maximum vocab size. Received vocabulary size is {new_vocab_size}; `max_tokens` is {self.max_tokens}.

What it means

The vocabulary plus reserved special tokens exceeds the layer's `max_tokens` cap, so the lookup table cannot hold it. set_vocabulary reports both the required size and the cap.

Source

Thrown at keras/src/layers/preprocessing/index_lookup.py:550

        if (
            self.oov_token is not None
            and self.invert
            and self.oov_token in tokens
        ):
            oov_index = np.argwhere(vocabulary == self.oov_token)[-1]
            raise ValueError(
                "Found reserved OOV token at unexpected location in "
                "`vocabulary`. Note that passed `vocabulary` does not need to "
                "include the OOV and mask tokens. Either remove all mask and "
                "OOV tokens, or include them only at the start of the "
                f"vocabulary in precisely this order: {special_tokens}. "
                f"Received: oov_token={self.oov_token} at "
                f"vocabulary index {oov_index}"
            )

        new_vocab_size = token_start + len(tokens)
        if self.max_tokens is not None and (new_vocab_size > self.max_tokens):
            raise ValueError(
                "Attempted to set a vocabulary larger than the maximum vocab "
                f"size. Received vocabulary size is {new_vocab_size}; "
                f"`max_tokens` is {self.max_tokens}."
            )
        self.lookup_table = self._lookup_table_from_tokens(tokens)
        self._record_vocabulary_size()

        if self.output_mode == "tf_idf" and idf_weights is not None:
            if len(vocabulary) != len(idf_weights):
                raise ValueError(
                    "`idf_weights` must be the same length as vocabulary. "
                    f"len(idf_weights) is {len(idf_weights)}; "
                    f"len(vocabulary) is {len(vocabulary)}"
                )
            idf_weights = self._convert_to_ndarray(idf_weights)
            if idf_weights.ndim != 1:
                raise ValueError(
                    "TF-IDF data must be a 1-index array. "

View on GitHub (pinned to 7a34a03db6)

Solutions

1. Raise max_tokens to at least `len(vocab) + num_oov_indices` (+1 more if a mask token is used).
2. Or prune the vocabulary to `max_tokens - reserved_slots`.
3. Compute the cap from data at build time instead of hard-coding it.

Example fix

# before
layer = IndexLookup(max_tokens=10000)
layer.set_vocabulary(vocab)  # len(vocab) + 1 > 10000

# after
layer = IndexLookup(max_tokens=len(vocab) + 2)  # OOV + mask slots
layer.set_vocabulary(vocab)

reserved = num_oov + (1 if mask_token else 0)
if len(vocab) + reserved > max_tokens:
    raise ValueError('vocab %d + %d reserved exceeds max_tokens %d' % (len(vocab), reserved, max_tokens))
layer.set_vocabulary(vocab)

When it happens

Trigger: `set_vocabulary(vocab)` where `len(vocab) + reserved_slots > max_tokens`; also when passing `vocabulary=` in the constructor, since max_tokens counts the specials.

Common situations: Setting max_tokens equal to the observed vocabulary size and forgetting the +1 OOV (and +1 mask) slots; vocabulary growth between experiments with a fixed cap.

Related errors

• Vocabulary file {vocabulary} does not exist.
• Cannot set an empty vocabulary. Received: vocabulary={vocabu
• The passed vocabulary has at least one repeated term. Please
• Found reserved mask token at unexpected location in `vocabul
• Found reserved OOV token at unexpected location in `vocabula

AI-assisted analysis of keras-team/keras@7a34a03db6 (2026-08-25). Data as JSON: /api/errors/74d25ba08bf7b4e5. Report an issue: GitHub.