{"record":{"id":"1a1f3add84aa8d4d","repo":"dotnet/machinelearning","slug":"trying-to-merge-a-token-mergevalues-b-which-no","errorCode":null,"errorMessage":"Trying to merge a token '{mergeValues.b}' which not exist in the vocabulary.","messagePattern":"Trying to merge a token '(.+?)' which not exist in the vocabulary\\.","errorType":"exception","errorClass":"InvalidOperationException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs","lineNumber":375,"sourceCode":"            }\n\n            UnknownToken = unknownToken;\n\n            int prefixLen = ContinuingSubwordPrefix is null ? 0 : ContinuingSubwordPrefix.Length;\n\n            Merges = new();\n            for (int i = 0; i < merges.Count; i++)\n            {\n                (string a, string b) mergeValues = merges[i];\n\n                if (!_vocab.TryGetValue(mergeValues.a, out int aId))\n                {\n                    throw new InvalidOperationException($\"Trying to merge a token '{mergeValues.a}' which not exist in the vocabulary.\");\n                }\n\n                if (!_vocab.TryGetValue(mergeValues.b, out int bId))\n                {\n                    throw new InvalidOperationException($\"Trying to merge a token '{mergeValues.b}' which not exist in the vocabulary.\");\n                }\n\n                if (mergeValues.b.Length <= prefixLen)\n                {\n                    throw new InvalidOperationException($\"The merge value '{mergeValues.b}' is too short to be merged with a prefix of length {prefixLen}. This implies that the merge file is either damaged or missing the prefix in its entries.\");\n                }\n\n                string newToken = $\"{mergeValues.a}{mergeValues.b.Substring(prefixLen)}\";\n                if (!_vocab.TryGetValue(newToken, out int newId))\n                {\n                    throw new InvalidOperationException($\"Trying to merge a token '{newToken}' which not exist in the vocabulary.\");\n                }\n\n                Merges.Add(new Pair<int>(aId, bId), (i, newId));\n            }\n        }\n\n        /// <summary>","sourceCodeStart":357,"sourceCodeEnd":393,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs#L357-L393","documentation":"The same merge validation as the previous error, but for the second ('b') element of a merge pair: it must exist in the vocabulary or the BpeTokenizer constructor throws InvalidOperationException. This keeps the merge table consistent with the token-id space.","triggerScenarios":"A merges.txt entry whose right-hand token is absent from vocab.json — mismatched model files, a truncated vocab, or manually appended merge rules.","commonSituations":"Mixing vocab/merges revisions; hand-editing merges.txt to add custom merges without adding the corresponding vocab entries; partial downloads of model artifacts.","solutions":["Load vocab.json and merges.txt from the same model release/revision.","Add any missing tokens to the vocabulary if the merges are intentionally custom.","Re-download and verify the model files (checksum comparison).","Remove hand-added merge lines whose tokens were never added to the vocab."],"exampleFix":"// before\n// merges.txt has custom line 'he llo' but 'llo' is not in vocab\nvar tok = new BpeTokenizer(vocab, merges);\n// after\nvocab[\"llo\"] = vocab.Count; // add token used by the custom merge\nvar tok = new BpeTokenizer(vocab, merges);","handlingStrategy":"validation","validationCode":"foreach (var (_, b) in merges)\n    if (!vocab.ContainsKey(b)) throw new InvalidOperationException($\"Merge token '{b}' missing from vocab; vocab/merges files are mismatched\");","typeGuard":"null","tryCatchPattern":"try { var tok = new BpeTokenizer(vocab, merges); }\ncatch (InvalidOperationException ex) when (ex.Message.StartsWith(\"Trying to merge a token\")) { throw new InvalidDataException(\"merges.txt references tokens absent from vocab.json\", ex); }","preventionTips":["Pin model artifacts to a single revision (same commit/tag).","Preload-validate the whole merge table before constructing the tokenizer.","Re-download files if any merge entry fails vocab lookup."],"tags":["vocabulary","merge-file","bpe","model-files"],"backgroundTag":"entity-not-found","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}