dotnet/machinelearning · error · InvalidOperationException

Trying to merge a token '{mergeValues.b}' which not exist in

Error message

Trying to merge a token '{mergeValues.b}' which not exist in the vocabulary.

What it means

The same merge validation as the previous error, but for the second ('b') element of a merge pair: it must exist in the vocabulary or the BpeTokenizer constructor throws InvalidOperationException. This keeps the merge table consistent with the token-id space.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs:375

            }

            UnknownToken = unknownToken;

            int prefixLen = ContinuingSubwordPrefix is null ? 0 : ContinuingSubwordPrefix.Length;

            Merges = new();
            for (int i = 0; i < merges.Count; i++)
            {
                (string a, string b) mergeValues = merges[i];

                if (!_vocab.TryGetValue(mergeValues.a, out int aId))
                {
                    throw new InvalidOperationException($"Trying to merge a token '{mergeValues.a}' which not exist in the vocabulary.");
                }

                if (!_vocab.TryGetValue(mergeValues.b, out int bId))
                {
                    throw new InvalidOperationException($"Trying to merge a token '{mergeValues.b}' which not exist in the vocabulary.");
                }

                if (mergeValues.b.Length <= prefixLen)
                {
                    throw new InvalidOperationException($"The merge value '{mergeValues.b}' is too short to be merged with a prefix of length {prefixLen}. This implies that the merge file is either damaged or missing the prefix in its entries.");
                }

                string newToken = $"{mergeValues.a}{mergeValues.b.Substring(prefixLen)}";
                if (!_vocab.TryGetValue(newToken, out int newId))
                {
                    throw new InvalidOperationException($"Trying to merge a token '{newToken}' which not exist in the vocabulary.");
                }

                Merges.Add(new Pair<int>(aId, bId), (i, newId));
            }
        }

        /// <summary>

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Load vocab.json and merges.txt from the same model release/revision.
2. Add any missing tokens to the vocabulary if the merges are intentionally custom.
3. Re-download and verify the model files (checksum comparison).
4. Remove hand-added merge lines whose tokens were never added to the vocab.

Example fix

// before
// merges.txt has custom line 'he llo' but 'llo' is not in vocab
var tok = new BpeTokenizer(vocab, merges);
// after
vocab["llo"] = vocab.Count; // add token used by the custom merge
var tok = new BpeTokenizer(vocab, merges);

foreach (var (_, b) in merges)
    if (!vocab.ContainsKey(b)) throw new InvalidOperationException($"Merge token '{b}' missing from vocab; vocab/merges files are mismatched");

null

try { var tok = new BpeTokenizer(vocab, merges); }
catch (InvalidOperationException ex) when (ex.Message.StartsWith("Trying to merge a token")) { throw new InvalidDataException("merges.txt references tokens absent from vocab.json", ex); }

When it happens

Trigger: A merges.txt entry whose right-hand token is absent from vocab.json — mismatched model files, a truncated vocab, or manually appended merge rules.

Common situations: Mixing vocab/merges revisions; hand-editing merges.txt to add custom merges without adding the corresponding vocab entries; partial downloads of model artifacts.

Understand the failure class

Background: "Not found" and "does not exist" errors: why "Task not found", "No such folder", and "Can't find" fire when a lookup comes back empty — this error's family across 14 libraries.

Related errors

• Trying to merge a token '{mergeValues.a}' which not exist in
• Trying to merge a token '{newToken}' which not exist in the
• Unknown Token '{value}' was not present in '{nameof(Vocabula
• The vocabulary cannot be null.
• The beginning of sentence token '{beginningOfSentenceToken}'

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/1a1f3add84aa8d4d. Report an issue: GitHub.