{"record":{"id":"74d25ba08bf7b4e5","repo":"keras-team/keras","slug":"attempted-to-set-a-vocabulary-larger-than-the-maxi","errorCode":null,"errorMessage":"Attempted to set a vocabulary larger than the maximum vocab size. Received vocabulary size is {new_vocab_size}; `max_tokens` is {self.max_tokens}.","messagePattern":"Attempted to set a vocabulary larger than the maximum vocab size\\. Received vocabulary size is (.+?); `max_tokens` is (.+?)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"keras/src/layers/preprocessing/index_lookup.py","lineNumber":550,"sourceCode":"        if (\n            self.oov_token is not None\n            and self.invert\n            and self.oov_token in tokens\n        ):\n            oov_index = np.argwhere(vocabulary == self.oov_token)[-1]\n            raise ValueError(\n                \"Found reserved OOV token at unexpected location in \"\n                \"`vocabulary`. Note that passed `vocabulary` does not need to \"\n                \"include the OOV and mask tokens. Either remove all mask and \"\n                \"OOV tokens, or include them only at the start of the \"\n                f\"vocabulary in precisely this order: {special_tokens}. \"\n                f\"Received: oov_token={self.oov_token} at \"\n                f\"vocabulary index {oov_index}\"\n            )\n\n        new_vocab_size = token_start + len(tokens)\n        if self.max_tokens is not None and (new_vocab_size > self.max_tokens):\n            raise ValueError(\n                \"Attempted to set a vocabulary larger than the maximum vocab \"\n                f\"size. Received vocabulary size is {new_vocab_size}; \"\n                f\"`max_tokens` is {self.max_tokens}.\"\n            )\n        self.lookup_table = self._lookup_table_from_tokens(tokens)\n        self._record_vocabulary_size()\n\n        if self.output_mode == \"tf_idf\" and idf_weights is not None:\n            if len(vocabulary) != len(idf_weights):\n                raise ValueError(\n                    \"`idf_weights` must be the same length as vocabulary. \"\n                    f\"len(idf_weights) is {len(idf_weights)}; \"\n                    f\"len(vocabulary) is {len(vocabulary)}\"\n                )\n            idf_weights = self._convert_to_ndarray(idf_weights)\n            if idf_weights.ndim != 1:\n                raise ValueError(\n                    \"TF-IDF data must be a 1-index array. \"","sourceCodeStart":532,"sourceCodeEnd":568,"githubUrl":"https://github.com/keras-team/keras/blob/7a34a03db60bf60042242d6a556fc3be119046a5/keras/src/layers/preprocessing/index_lookup.py#L532-L568","documentation":"The vocabulary plus reserved special tokens exceeds the layer's `max_tokens` cap, so the lookup table cannot hold it. set_vocabulary reports both the required size and the cap.","triggerScenarios":"`set_vocabulary(vocab)` where `len(vocab) + reserved_slots > max_tokens`; also when passing `vocabulary=` in the constructor, since max_tokens counts the specials.","commonSituations":"Setting max_tokens equal to the observed vocabulary size and forgetting the +1 OOV (and +1 mask) slots; vocabulary growth between experiments with a fixed cap.","solutions":["Raise max_tokens to at least `len(vocab) + num_oov_indices` (+1 more if a mask token is used).","Or prune the vocabulary to `max_tokens - reserved_slots`.","Compute the cap from data at build time instead of hard-coding it."],"exampleFix":"# before\nlayer = IndexLookup(max_tokens=10000)\nlayer.set_vocabulary(vocab)  # len(vocab) + 1 > 10000\n\n# after\nlayer = IndexLookup(max_tokens=len(vocab) + 2)  # OOV + mask slots\nlayer.set_vocabulary(vocab)","handlingStrategy":"validation","validationCode":"reserved = num_oov + (1 if mask_token else 0)\nif len(vocab) + reserved > max_tokens:\n    raise ValueError('vocab %d + %d reserved exceeds max_tokens %d' % (len(vocab), reserved, max_tokens))\nlayer.set_vocabulary(vocab)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Budget max_tokens as expected_vocab_size plus OOV and mask slots.","Recompute the cap whenever the vocabulary is regenerated."],"tags":["keras","preprocessing","index-lookup","vocabulary","size-limit"],"backgroundTag":"size-limit-exceeded","analyzedSha":"7a34a03db60bf60042242d6a556fc3be119046a5","analyzedAt":"2026-08-25T21:25:25.994Z","schemaVersion":2},"datasetVersion":"2026-08-26T02:17:13.382Z"}