{"record":{"id":"f1a08c3766e5fc41","repo":"keras-team/keras","slug":"the-passed-vocabulary-has-at-least-one-repeated-te","errorCode":null,"errorMessage":"The passed vocabulary has at least one repeated term. Please uniquify your dataset. The repeated terms are: {repeated_tokens}","messagePattern":"The passed vocabulary has at least one repeated term\\. Please uniquify your dataset\\. The repeated terms are: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"keras/src/layers/preprocessing/index_lookup.py","lineNumber":513,"sourceCode":"                f\"Received: vocabulary={vocabulary}\"\n            )\n\n        oov_start = self._oov_start_index()\n        token_start = self._token_start_index()\n        special_tokens = [self.mask_token] * oov_start + [\n            self.oov_token\n        ] * self.num_oov_indices\n        found_special_tokens = np.array_equal(\n            special_tokens, vocabulary[:token_start]\n        )\n        if found_special_tokens:\n            tokens = vocabulary[token_start:]\n        else:\n            tokens = vocabulary\n\n        repeated_tokens = self._find_repeated_tokens(tokens)\n        if repeated_tokens:\n            raise ValueError(\n                \"The passed vocabulary has at least one repeated \"\n                \"term. Please uniquify your dataset. The repeated terms \"\n                f\"are: {repeated_tokens}\"\n            )\n\n        if self.mask_token is not None and self.mask_token in tokens:\n            mask_index = np.argwhere(vocabulary == self.mask_token)[-1]\n            raise ValueError(\n                \"Found reserved mask token at unexpected location in \"\n                \"`vocabulary`. Note that passed `vocabulary` does not need to \"\n                \"include the OOV and mask tokens. Either remove all mask and \"\n                \"OOV tokens, or include them only at the start of the \"\n                f\"vocabulary in precisely this order: {special_tokens}. \"\n                f\"Received: mask_token={self.mask_token} at \"\n                f\"vocabulary index {mask_index}\"\n            )\n        # Only error out for oov_token when invert=True. When invert=False,\n        # oov_token is unused during lookup.","sourceCodeStart":495,"sourceCodeEnd":531,"githubUrl":"https://github.com/keras-team/keras/blob/7a34a03db60bf60042242d6a556fc3be119046a5/keras/src/layers/preprocessing/index_lookup.py#L495-L531","documentation":"IndexLookup maps each vocabulary term to a unique index, so the vocabulary must contain no duplicates. This ValueError fires from set_vocabulary() when the supplied (or adapted) vocabulary contains repeated terms; the message lists the offending duplicates.","triggerScenarios":"`set_vocabulary(['the', 'the', 'cat'])`, or concatenating per-shard vocabulary files that overlap.","commonSituations":"Merging vocabularies from distributed training shards without dedup; normalization (lowercasing, stripping) collapsing distinct lines to the same token; vocab files with duplicate lines.","solutions":["Dedupe while preserving order: `vocab = list(dict.fromkeys(vocab))`.","After merging vocab files, always run a dedupe pass.","Check for near-duplicates introduced by your normalization pipeline."],"exampleFix":"# before\nlayer.set_vocabulary(vocab)\n\n# after\nvocab = list(dict.fromkeys(vocab))  # order-preserving dedupe\nlayer.set_vocabulary(vocab)","handlingStrategy":"validation","validationCode":"if len(set(vocab)) != len(vocab):\n    vocab = list(dict.fromkeys(vocab))\nlayer.set_vocabulary(vocab)","typeGuard":"def is_unique(seq) -> bool:\n    return len(set(seq)) == len(seq)","tryCatchPattern":null,"preventionTips":["Dedupe every merged or loaded vocabulary with dict.fromkeys before use.","Apply the same normalization the layer will use before checking uniqueness."],"tags":["keras","preprocessing","index-lookup","vocabulary","duplicates"],"backgroundTag":"duplicate-values","analyzedSha":"7a34a03db60bf60042242d6a556fc3be119046a5","analyzedAt":"2026-08-25T21:25:25.994Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}