dotnet/machinelearning · error · InvalidOperationException

Trying to merge a token '{mergeValues.a}' which not exist in

Error message

Trying to merge a token '{mergeValues.a}' which not exist in the vocabulary.

What it means

During construction the BpeTokenizer validates every merge pair from the merges file: the first ('a') element of each merge must exist in the vocabulary. If it does not, the constructor throws InvalidOperationException because merge rules referencing unknown tokens would corrupt tokenization.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs:370

            if (specialTokens is not null)
            {
                SpecialTokens = specialTokens;
                _specialTokens = specialTokens.ToDictionary(kvp => new StringSpanOrdinalKey(kvp.Key), kvp => (kvp.Value, kvp.Key));
                _specialTokensReverse = specialTokens.ToDictionary(kvp => kvp.Value, kvp => kvp.Key);
            }

            UnknownToken = unknownToken;

            int prefixLen = ContinuingSubwordPrefix is null ? 0 : ContinuingSubwordPrefix.Length;

            Merges = new();
            for (int i = 0; i < merges.Count; i++)
            {
                (string a, string b) mergeValues = merges[i];

                if (!_vocab.TryGetValue(mergeValues.a, out int aId))
                {
                    throw new InvalidOperationException($"Trying to merge a token '{mergeValues.a}' which not exist in the vocabulary.");
                }

                if (!_vocab.TryGetValue(mergeValues.b, out int bId))
                {
                    throw new InvalidOperationException($"Trying to merge a token '{mergeValues.b}' which not exist in the vocabulary.");
                }

                if (mergeValues.b.Length <= prefixLen)
                {
                    throw new InvalidOperationException($"The merge value '{mergeValues.b}' is too short to be merged with a prefix of length {prefixLen}. This implies that the merge file is either damaged or missing the prefix in its entries.");
                }

                string newToken = $"{mergeValues.a}{mergeValues.b.Substring(prefixLen)}";
                if (!_vocab.TryGetValue(newToken, out int newId))
                {
                    throw new InvalidOperationException($"Trying to merge a token '{newToken}' which not exist in the vocabulary.");
                }

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Use the vocab.json and merges.txt that ship together from the same model release.
2. Check file integrity/checksums — re-download the model files.
3. Ensure the vocab includes all single-character and prefix tokens the merges rely on.
4. If a merge is genuinely stale, remove that line from the merges file (only if you control it).

Example fix

// before
var vocab = LoadJson("vocab_b.json");   // wrong revision
var merges = LoadMerges("merges_a.txt"); // from another revision
var tok = new BpeTokenizer(vocab, merges);
// after
// both files from the same model snapshot
var vocab = LoadJson("vocab.json");
var merges = LoadMerges("merges.txt");
var tok = new BpeTokenizer(vocab, merges);

foreach (var (a, _) in merges)
    if (!vocab.ContainsKey(a)) throw new InvalidOperationException($"Merge token '{a}' missing from vocab; vocab/merges files are mismatched");

null

try { var tok = new BpeTokenizer(vocab, merges); }
catch (InvalidOperationException ex) when (ex.Message.StartsWith("Trying to merge a token")) { throw new InvalidDataException("vocab.json and merges.txt do not match; use files from the same model release", ex); }

When it happens

Trigger: Loading a vocab.json and merges.txt from mismatched sources (e.g. vocab from model A, merges from model B), or a vocab file missing entries the merges file expects.

Common situations: Downloading vocab and merges from different model revisions; manually editing/truncating vocab.json; GPT-2 style merges whose left token was never added to the custom vocab; files from incompatible tokenizer versions.

Understand the failure class

Background: "Not found" and "does not exist" errors: why "Task not found", "No such folder", and "Can't find" fire when a lookup comes back empty — this error's family across 14 libraries.

Related errors

• Trying to merge a token '{mergeValues.b}' which not exist in
• Trying to merge a token '{newToken}' which not exist in the
• Unknown Token '{value}' was not present in '{nameof(Vocabula
• The vocabulary cannot be null.
• The beginning of sentence token '{beginningOfSentenceToken}'

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/e86954935d2685f8. Report an issue: GitHub.