keras-team/keras · error · ValueError

The passed vocabulary has at least one repeated term. Please

Error message

The passed vocabulary has at least one repeated term. Please uniquify your dataset. The repeated terms are: {repeated_tokens}

What it means

A lookup table requires unique keys; duplicate tokens would make the token-to-index mapping ambiguous. set_vocabulary detects repeats and reports the offending tokens so you can uniquify.

Source

Thrown at keras/src/layers/preprocessing/index_lookup.py:513

                f"Received: vocabulary={vocabulary}"
            )

        oov_start = self._oov_start_index()
        token_start = self._token_start_index()
        special_tokens = [self.mask_token] * oov_start + [
            self.oov_token
        ] * self.num_oov_indices
        found_special_tokens = np.array_equal(
            special_tokens, vocabulary[:token_start]
        )
        if found_special_tokens:
            tokens = vocabulary[token_start:]
        else:
            tokens = vocabulary

        repeated_tokens = self._find_repeated_tokens(tokens)
        if repeated_tokens:
            raise ValueError(
                "The passed vocabulary has at least one repeated "
                "term. Please uniquify your dataset. The repeated terms "
                f"are: {repeated_tokens}"
            )

        if self.mask_token is not None and self.mask_token in tokens:
            mask_index = np.argwhere(vocabulary == self.mask_token)[-1]
            raise ValueError(
                "Found reserved mask token at unexpected location in "
                "`vocabulary`. Note that passed `vocabulary` does not need to "
                "include the OOV and mask tokens. Either remove all mask and "
                "OOV tokens, or include them only at the start of the "
                f"vocabulary in precisely this order: {special_tokens}. "
                f"Received: mask_token={self.mask_token} at "
                f"vocabulary index {mask_index}"
            )
        # Only error out for oov_token when invert=True. When invert=False,
        # oov_token is unused during lookup.

View on GitHub (pinned to 7a34a03db6)

Solutions

Dedupe while preserving order: `vocab = list(dict.fromkeys(vocab))`.
After merging vocab files, always run a dedupe pass.
Check for near-duplicates introduced by your normalization pipeline.

Example fix

# before
layer.set_vocabulary(vocab)

# after
vocab = list(dict.fromkeys(vocab))  # order-preserving dedupe
layer.set_vocabulary(vocab)

Defensive patterns

Strategy: validation

Validate before calling

if len(set(vocab)) != len(vocab):
    vocab = list(dict.fromkeys(vocab))
layer.set_vocabulary(vocab)

Type guard

def is_unique(seq) -> bool:
    return len(set(seq)) == len(seq)

Prevention

Dedupe every merged or loaded vocabulary with dict.fromkeys before use.
Apply the same normalization the layer will use before checking uniqueness.

When it happens

Trigger: `set_vocabulary(['the', 'the', 'cat'])`, or concatenating per-shard vocabulary files that overlap.

Common situations: Merging vocabularies from distributed training shards without dedup; normalization (lowercasing, stripping) collapsing distinct lines to the same token; vocab files with duplicate lines.

Related errors

AI-assisted analysis of keras-team/keras@7a34a03db6 (2026-08-25). Data as JSON: /api/errors/f1a08c3766e5fc41. Report an issue: GitHub.