{"record":{"id":"e86954935d2685f8","repo":"dotnet/machinelearning","slug":"trying-to-merge-a-token-mergevalues-a-which-no","errorCode":null,"errorMessage":"Trying to merge a token '{mergeValues.a}' which not exist in the vocabulary.","messagePattern":"Trying to merge a token '(.+?)' which not exist in the vocabulary\\.","errorType":"exception","errorClass":"InvalidOperationException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs","lineNumber":370,"sourceCode":"            if (specialTokens is not null)\n            {\n                SpecialTokens = specialTokens;\n                _specialTokens = specialTokens.ToDictionary(kvp => new StringSpanOrdinalKey(kvp.Key), kvp => (kvp.Value, kvp.Key));\n                _specialTokensReverse = specialTokens.ToDictionary(kvp => kvp.Value, kvp => kvp.Key);\n            }\n\n            UnknownToken = unknownToken;\n\n            int prefixLen = ContinuingSubwordPrefix is null ? 0 : ContinuingSubwordPrefix.Length;\n\n            Merges = new();\n            for (int i = 0; i < merges.Count; i++)\n            {\n                (string a, string b) mergeValues = merges[i];\n\n                if (!_vocab.TryGetValue(mergeValues.a, out int aId))\n                {\n                    throw new InvalidOperationException($\"Trying to merge a token '{mergeValues.a}' which not exist in the vocabulary.\");\n                }\n\n                if (!_vocab.TryGetValue(mergeValues.b, out int bId))\n                {\n                    throw new InvalidOperationException($\"Trying to merge a token '{mergeValues.b}' which not exist in the vocabulary.\");\n                }\n\n                if (mergeValues.b.Length <= prefixLen)\n                {\n                    throw new InvalidOperationException($\"The merge value '{mergeValues.b}' is too short to be merged with a prefix of length {prefixLen}. This implies that the merge file is either damaged or missing the prefix in its entries.\");\n                }\n\n                string newToken = $\"{mergeValues.a}{mergeValues.b.Substring(prefixLen)}\";\n                if (!_vocab.TryGetValue(newToken, out int newId))\n                {\n                    throw new InvalidOperationException($\"Trying to merge a token '{newToken}' which not exist in the vocabulary.\");\n                }\n","sourceCodeStart":352,"sourceCodeEnd":388,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs#L352-L388","documentation":"During construction the BpeTokenizer validates every merge pair from the merges file: the first ('a') element of each merge must exist in the vocabulary. If it does not, the constructor throws InvalidOperationException because merge rules referencing unknown tokens would corrupt tokenization.","triggerScenarios":"Loading a vocab.json and merges.txt from mismatched sources (e.g. vocab from model A, merges from model B), or a vocab file missing entries the merges file expects.","commonSituations":"Downloading vocab and merges from different model revisions; manually editing/truncating vocab.json; GPT-2 style merges whose left token was never added to the custom vocab; files from incompatible tokenizer versions.","solutions":["Use the vocab.json and merges.txt that ship together from the same model release.","Check file integrity/checksums — re-download the model files.","Ensure the vocab includes all single-character and prefix tokens the merges rely on.","If a merge is genuinely stale, remove that line from the merges file (only if you control it)."],"exampleFix":"// before\nvar vocab = LoadJson(\"vocab_b.json\");   // wrong revision\nvar merges = LoadMerges(\"merges_a.txt\"); // from another revision\nvar tok = new BpeTokenizer(vocab, merges);\n// after\n// both files from the same model snapshot\nvar vocab = LoadJson(\"vocab.json\");\nvar merges = LoadMerges(\"merges.txt\");\nvar tok = new BpeTokenizer(vocab, merges);","handlingStrategy":"validation","validationCode":"foreach (var (a, _) in merges)\n    if (!vocab.ContainsKey(a)) throw new InvalidOperationException($\"Merge token '{a}' missing from vocab; vocab/merges files are mismatched\");","typeGuard":"null","tryCatchPattern":"try { var tok = new BpeTokenizer(vocab, merges); }\ncatch (InvalidOperationException ex) when (ex.Message.StartsWith(\"Trying to merge a token\")) { throw new InvalidDataException(\"vocab.json and merges.txt do not match; use files from the same model release\", ex); }","preventionTips":["Always download vocab.json and merges.txt together from one model snapshot.","Verify checksums of model artifacts before loading.","Never hand-edit merges without updating the vocab."],"tags":["vocabulary","merge-file","bpe","model-files"],"backgroundTag":"entity-not-found","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}