{"record":{"id":"f1a08c3766e5fc41","repo":"keras-team/keras","slug":"the-passed-vocabulary-has-at-least-one-repeated-te","errorCode":null,"errorMessage":"The passed vocabulary has at least one repeated term. Please uniquify your dataset. The repeated terms are: {repeated_tokens}","messagePattern":"The passed vocabulary has at least one repeated term\\. Please uniquify your dataset\\. The repeated terms are: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"keras/src/layers/preprocessing/index_lookup.py","lineNumber":513,"sourceCode":"                f\"Received: vocabulary={vocabulary}\"\n            )\n\n        oov_start = self._oov_start_index()\n        token_start = self._token_start_index()\n        special_tokens = [self.mask_token] * oov_start + [\n            self.oov_token\n        ] * self.num_oov_indices\n        found_special_tokens = np.array_equal(\n            special_tokens, vocabulary[:token_start]\n        )\n        if found_special_tokens:\n            tokens = vocabulary[token_start:]\n        else:\n            tokens = vocabulary\n\n        repeated_tokens = self._find_repeated_tokens(tokens)\n        if repeated_tokens:\n            raise ValueError(\n                \"The passed vocabulary has at least one repeated \"\n                \"term. Please uniquify your dataset. The repeated terms \"\n                f\"are: {repeated_tokens}\"\n            )\n\n        if self.mask_token is not None and self.mask_token in tokens:\n            mask_index = np.argwhere(vocabulary == self.mask_token)[-1]\n            raise ValueError(\n                \"Found reserved mask token at unexpected location in \"\n                \"`vocabulary`. Note that passed `vocabulary` does not need to \"\n                \"include the OOV and mask tokens. Either remove all mask and \"\n                \"OOV tokens, or include them only at the start of the \"\n                f\"vocabulary in precisely this order: {special_tokens}. \"\n                f\"Received: mask_token={self.mask_token} at \"\n                f\"vocabulary index {mask_index}\"\n            )\n        # Only error out for oov_token when invert=True. When invert=False,\n        # oov_token is unused during lookup.","sourceCodeStart":495,"sourceCodeEnd":531,"githubUrl":"https://github.com/keras-team/keras/blob/7a34a03db60bf60042242d6a556fc3be119046a5/keras/src/layers/preprocessing/index_lookup.py#L495-L531","documentation":"A lookup table requires unique keys; duplicate tokens would make the token-to-index mapping ambiguous. set_vocabulary detects repeats and reports the offending tokens so you can uniquify.","triggerScenarios":"`set_vocabulary(['the', 'the', 'cat'])`, or concatenating per-shard vocabulary files that overlap.","commonSituations":"Merging vocabularies from distributed training shards without dedup; normalization (lowercasing, stripping) collapsing distinct lines to the same token; vocab files with duplicate lines.","solutions":["Dedupe while preserving order: `vocab = list(dict.fromkeys(vocab))`.","After merging vocab files, always run a dedupe pass.","Check for near-duplicates introduced by your normalization pipeline."],"exampleFix":"# before\nlayer.set_vocabulary(vocab)\n\n# after\nvocab = list(dict.fromkeys(vocab))  # order-preserving dedupe\nlayer.set_vocabulary(vocab)","handlingStrategy":"validation","validationCode":"if len(set(vocab)) != len(vocab):\n    vocab = list(dict.fromkeys(vocab))\nlayer.set_vocabulary(vocab)","typeGuard":"def is_unique(seq) -> bool:\n    return len(set(seq)) == len(seq)","tryCatchPattern":null,"preventionTips":["Dedupe every merged or loaded vocabulary with dict.fromkeys before use.","Apply the same normalization the layer will use before checking uniqueness."],"tags":["keras","preprocessing","index-lookup","vocabulary","duplicates"],"backgroundTag":"duplicate-values","analyzedSha":"7a34a03db60bf60042242d6a556fc3be119046a5","analyzedAt":"2026-08-25T21:25:25.994Z","schemaVersion":2},"datasetVersion":"2026-08-26T02:17:13.382Z"}