dotnet/machinelearning · error · InvalidOperationException

Trying to merge a token '{newToken}' which not exist in the

Error message

Trying to merge a token '{newToken}' which not exist in the vocabulary.

What it means

After combining the two sides of a merge (accounting for the subword prefix), the resulting merged token must itself exist in the vocabulary; otherwise the constructor throws InvalidOperationException. A merge rule whose output token is not in the vocab would produce ids that cannot be decoded or reused.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs:386

                if (!_vocab.TryGetValue(mergeValues.a, out int aId))
                {
                    throw new InvalidOperationException($"Trying to merge a token '{mergeValues.a}' which not exist in the vocabulary.");
                }

                if (!_vocab.TryGetValue(mergeValues.b, out int bId))
                {
                    throw new InvalidOperationException($"Trying to merge a token '{mergeValues.b}' which not exist in the vocabulary.");
                }

                if (mergeValues.b.Length <= prefixLen)
                {
                    throw new InvalidOperationException($"The merge value '{mergeValues.b}' is too short to be merged with a prefix of length {prefixLen}. This implies that the merge file is either damaged or missing the prefix in its entries.");
                }

                string newToken = $"{mergeValues.a}{mergeValues.b.Substring(prefixLen)}";
                if (!_vocab.TryGetValue(newToken, out int newId))
                {
                    throw new InvalidOperationException($"Trying to merge a token '{newToken}' which not exist in the vocabulary.");
                }

                Merges.Add(new Pair<int>(aId, bId), (i, newId));
            }
        }

        /// <summary>
        /// Gets a value indicating whether to handle the input text in byte level.
        /// if true, the input text will be converted to UTF-8 bytes before encoding it.
        /// Additionally, some ASCII characters will be transformed to another characters (e.g Space character will be transformed to 'Ġ' character).
        /// </summary>
        public bool ByteLevel { get; }

        /// <summary>
        /// Gets the optional beginning of sentence token.
        /// </summary>
        public string? BeginningOfSentenceToken { get; }

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Use vocab.json and merges.txt from the same complete model export.
2. If pruning the vocab, also prune (or skip) merge rules whose merged token was removed.
3. Add the merged token to the vocabulary if the merge is intentionally custom.
4. Re-download model artifacts and validate file hashes.

Example fix

// before
// vocab pruned, but merges.txt still contains the merge producing 'llo'
var tok = new BpeTokenizer(vocab, merges);
// after
vocab["llo"] = vocab.Count; // keep merged tokens referenced by merges
var tok = new BpeTokenizer(vocab, merges);

foreach (var (a, b) in merges)
    if (!vocab.ContainsKey(a + b)) throw new InvalidOperationException($"Merged token '{a}{b}' missing from vocab; regenerate or re-download model files");

null

try { var tok = new BpeTokenizer(vocab, merges); }
catch (InvalidOperationException ex) when (ex.Message.StartsWith("Trying to merge a token")) { throw new InvalidDataException("Vocab is missing tokens produced by merge rules; use unpruned vocab.json", ex); }

When it happens

Trigger: A merges.txt entry (a, b) where the concatenated token a+b minus prefix is missing from vocab.json — mismatched vocab/merges revisions or a manually extended merges file.

Common situations: Vocab files pruned by frequency thresholding that dropped merged tokens still referenced by merges.txt; merging model files from different training checkpoints; custom merge additions without vocab updates.

Understand the failure class

Background: "Not found" and "does not exist" errors: why "Task not found", "No such folder", and "Can't find" fire when a lookup comes back empty — this error's family across 14 libraries.

Related errors

• Trying to merge a token '{mergeValues.a}' which not exist in
• Trying to merge a token '{mergeValues.b}' which not exist in
• Unknown Token '{value}' was not present in '{nameof(Vocabula
• The vocabulary cannot be null.
• The beginning of sentence token '{beginningOfSentenceToken}'

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/0408327b0faef5ac. Report an issue: GitHub.