{"record":{"id":"0b2f884c3074d69c","repo":"keras-team/keras","slug":"found-reserved-mask-token-at-unexpected-location-i","errorCode":null,"errorMessage":"Found reserved mask token at unexpected location in `vocabulary`. Note that passed `vocabulary` does not need to include the OOV and mask tokens. Either remove all mask and OOV tokens, or include them only at the start of the vocabulary in precisely this order: {special_tokens}. Received: mask_token={self.mask_token} at vocabulary index {mask_index}","messagePattern":"Found reserved mask token at unexpected location in `vocabulary`\\. Note that passed `vocabulary` does not need to include the OOV and mask tokens\\. Either remove all mask and OOV tokens, or include them only at the start of the vocabulary in precisely this order: (.+?)\\. Received: mask_token=(.+?) at vocabulary index (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"keras/src/layers/preprocessing/index_lookup.py","lineNumber":521,"sourceCode":"        found_special_tokens = np.array_equal(\n            special_tokens, vocabulary[:token_start]\n        )\n        if found_special_tokens:\n            tokens = vocabulary[token_start:]\n        else:\n            tokens = vocabulary\n\n        repeated_tokens = self._find_repeated_tokens(tokens)\n        if repeated_tokens:\n            raise ValueError(\n                \"The passed vocabulary has at least one repeated \"\n                \"term. Please uniquify your dataset. The repeated terms \"\n                f\"are: {repeated_tokens}\"\n            )\n\n        if self.mask_token is not None and self.mask_token in tokens:\n            mask_index = np.argwhere(vocabulary == self.mask_token)[-1]\n            raise ValueError(\n                \"Found reserved mask token at unexpected location in \"\n                \"`vocabulary`. Note that passed `vocabulary` does not need to \"\n                \"include the OOV and mask tokens. Either remove all mask and \"\n                \"OOV tokens, or include them only at the start of the \"\n                f\"vocabulary in precisely this order: {special_tokens}. \"\n                f\"Received: mask_token={self.mask_token} at \"\n                f\"vocabulary index {mask_index}\"\n            )\n        # Only error out for oov_token when invert=True. When invert=False,\n        # oov_token is unused during lookup.\n        if (\n            self.oov_token is not None\n            and self.invert\n            and self.oov_token in tokens\n        ):\n            oov_index = np.argwhere(vocabulary == self.oov_token)[-1]\n            raise ValueError(\n                \"Found reserved OOV token at unexpected location in \"","sourceCodeStart":503,"sourceCodeEnd":539,"githubUrl":"https://github.com/keras-team/keras/blob/7a34a03db60bf60042242d6a556fc3be119046a5/keras/src/layers/preprocessing/index_lookup.py#L503-L539","documentation":"IndexLookup reserves special tokens (mask token such as '', OOV token such as '[UNK]') at fixed head positions of the index space. If the mask token appears anywhere in the passed vocabulary other than its reserved head slot, the index mapping would conflict, so it is rejected with the expected token order spelled out.","triggerScenarios":"Passing a vocabulary that already contains the mask token (default '') anywhere — e.g. a vocabulary file exported from an older TextVectorization that included special tokens — while mask_token is set.","commonSituations":"Round-tripping vocabularies exported from fitted layers whose special-token layout (num_oov_indices, mask_token) differs from the new layer's settings.","solutions":["Strip special tokens before passing: `tokens = [t for t in vocab if t not in (mask_token, oov_token)]`.","Or include them only at the head, in exactly the documented order matching your mask_token, oov_token and num_oov_indices settings.","Set `mask_token=None` at construction if masking is genuinely not wanted."],"exampleFix":"# before\nlayer.set_vocabulary(vocab)  # vocab contains '' or '[UNK]'\n\n# after\ntokens = [t for t in vocab if t not in ('', '[UNK]')]\nlayer.set_vocabulary(tokens)","handlingStrategy":"validation","validationCode":"special = ('', '[UNK]')\ntokens = [t for t in vocab if t not in special]\nlayer.set_vocabulary(tokens)","typeGuard":"def free_of_specials(vocab, specials) -> bool:\n    return not any(t in specials for t in vocab)","tryCatchPattern":null,"preventionTips":["Export vocabularies without special tokens; layers add them automatically.","Match num_oov_indices and mask_token settings when round-tripping vocabularies between layers."],"tags":["keras","preprocessing","index-lookup","vocabulary","special-tokens"],"backgroundTag":"reserved-token-misplaced","analyzedSha":"7a34a03db60bf60042242d6a556fc3be119046a5","analyzedAt":"2026-08-25T21:25:25.994Z","schemaVersion":2},"datasetVersion":"2026-08-26T02:17:13.382Z"}