keras-team/keras · error · ValueError

Found reserved mask token at unexpected location in `vocabul

Error message

Found reserved mask token at unexpected location in `vocabulary`. Note that passed `vocabulary` does not need to include the OOV and mask tokens. Either remove all mask and OOV tokens, or include them only at the start of the vocabulary in precisely this order: {special_tokens}. Received: mask_token={self.mask_token} at vocabulary index {mask_index}

What it means

IndexLookup reserves special tokens (mask token such as '', OOV token such as '[UNK]') at fixed head positions of the index space. If the mask token appears anywhere in the passed vocabulary other than its reserved head slot, the index mapping would conflict, so it is rejected with the expected token order spelled out.

Source

Thrown at keras/src/layers/preprocessing/index_lookup.py:521

        found_special_tokens = np.array_equal(
            special_tokens, vocabulary[:token_start]
        )
        if found_special_tokens:
            tokens = vocabulary[token_start:]
        else:
            tokens = vocabulary

        repeated_tokens = self._find_repeated_tokens(tokens)
        if repeated_tokens:
            raise ValueError(
                "The passed vocabulary has at least one repeated "
                "term. Please uniquify your dataset. The repeated terms "
                f"are: {repeated_tokens}"
            )

        if self.mask_token is not None and self.mask_token in tokens:
            mask_index = np.argwhere(vocabulary == self.mask_token)[-1]
            raise ValueError(
                "Found reserved mask token at unexpected location in "
                "`vocabulary`. Note that passed `vocabulary` does not need to "
                "include the OOV and mask tokens. Either remove all mask and "
                "OOV tokens, or include them only at the start of the "
                f"vocabulary in precisely this order: {special_tokens}. "
                f"Received: mask_token={self.mask_token} at "
                f"vocabulary index {mask_index}"
            )
        # Only error out for oov_token when invert=True. When invert=False,
        # oov_token is unused during lookup.
        if (
            self.oov_token is not None
            and self.invert
            and self.oov_token in tokens
        ):
            oov_index = np.argwhere(vocabulary == self.oov_token)[-1]
            raise ValueError(
                "Found reserved OOV token at unexpected location in "

View on GitHub (pinned to 7a34a03db6)

Solutions

Strip special tokens before passing: `tokens = [t for t in vocab if t not in (mask_token, oov_token)]`.
Or include them only at the head, in exactly the documented order matching your mask_token, oov_token and num_oov_indices settings.
Set `mask_token=None` at construction if masking is genuinely not wanted.

Example fix

# before
layer.set_vocabulary(vocab)  # vocab contains '' or '[UNK]'

# after
tokens = [t for t in vocab if t not in ('', '[UNK]')]
layer.set_vocabulary(tokens)

Defensive patterns

Strategy: validation

Validate before calling

special = ('', '[UNK]')
tokens = [t for t in vocab if t not in special]
layer.set_vocabulary(tokens)

Type guard

def free_of_specials(vocab, specials) -> bool:
    return not any(t in specials for t in vocab)

Prevention

Export vocabularies without special tokens; layers add them automatically.
Match num_oov_indices and mask_token settings when round-tripping vocabularies between layers.

When it happens

Trigger: Passing a vocabulary that already contains the mask token (default '') anywhere — e.g. a vocabulary file exported from an older TextVectorization that included special tokens — while mask_token is set.

Common situations: Round-tripping vocabularies exported from fitted layers whose special-token layout (num_oov_indices, mask_token) differs from the new layer's settings.

Related errors

AI-assisted analysis of keras-team/keras@7a34a03db6 (2026-08-25). Data as JSON: /api/errors/0b2f884c3074d69c. Report an issue: GitHub.