{"record":{"id":"0408327b0faef5ac","repo":"dotnet/machinelearning","slug":"trying-to-merge-a-token-newtoken-which-not-exi","errorCode":null,"errorMessage":"Trying to merge a token '{newToken}' which not exist in the vocabulary.","messagePattern":"Trying to merge a token '(.+?)' which not exist in the vocabulary\\.","errorType":"exception","errorClass":"InvalidOperationException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs","lineNumber":386,"sourceCode":"                if (!_vocab.TryGetValue(mergeValues.a, out int aId))\n                {\n                    throw new InvalidOperationException($\"Trying to merge a token '{mergeValues.a}' which not exist in the vocabulary.\");\n                }\n\n                if (!_vocab.TryGetValue(mergeValues.b, out int bId))\n                {\n                    throw new InvalidOperationException($\"Trying to merge a token '{mergeValues.b}' which not exist in the vocabulary.\");\n                }\n\n                if (mergeValues.b.Length <= prefixLen)\n                {\n                    throw new InvalidOperationException($\"The merge value '{mergeValues.b}' is too short to be merged with a prefix of length {prefixLen}. This implies that the merge file is either damaged or missing the prefix in its entries.\");\n                }\n\n                string newToken = $\"{mergeValues.a}{mergeValues.b.Substring(prefixLen)}\";\n                if (!_vocab.TryGetValue(newToken, out int newId))\n                {\n                    throw new InvalidOperationException($\"Trying to merge a token '{newToken}' which not exist in the vocabulary.\");\n                }\n\n                Merges.Add(new Pair<int>(aId, bId), (i, newId));\n            }\n        }\n\n        /// <summary>\n        /// Gets a value indicating whether to handle the input text in byte level.\n        /// if true, the input text will be converted to UTF-8 bytes before encoding it.\n        /// Additionally, some ASCII characters will be transformed to another characters (e.g Space character will be transformed to 'Ġ' character).\n        /// </summary>\n        public bool ByteLevel { get; }\n\n        /// <summary>\n        /// Gets the optional beginning of sentence token.\n        /// </summary>\n        public string? BeginningOfSentenceToken { get; }\n","sourceCodeStart":368,"sourceCodeEnd":404,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs#L368-L404","documentation":"After combining the two sides of a merge (accounting for the subword prefix), the resulting merged token must itself exist in the vocabulary; otherwise the constructor throws InvalidOperationException. A merge rule whose output token is not in the vocab would produce ids that cannot be decoded or reused.","triggerScenarios":"A merges.txt entry (a, b) where the concatenated token a+b minus prefix is missing from vocab.json — mismatched vocab/merges revisions or a manually extended merges file.","commonSituations":"Vocab files pruned by frequency thresholding that dropped merged tokens still referenced by merges.txt; merging model files from different training checkpoints; custom merge additions without vocab updates.","solutions":["Use vocab.json and merges.txt from the same complete model export.","If pruning the vocab, also prune (or skip) merge rules whose merged token was removed.","Add the merged token to the vocabulary if the merge is intentionally custom.","Re-download model artifacts and validate file hashes."],"exampleFix":"// before\n// vocab pruned, but merges.txt still contains the merge producing 'llo'\nvar tok = new BpeTokenizer(vocab, merges);\n// after\nvocab[\"llo\"] = vocab.Count; // keep merged tokens referenced by merges\nvar tok = new BpeTokenizer(vocab, merges);","handlingStrategy":"validation","validationCode":"foreach (var (a, b) in merges)\n    if (!vocab.ContainsKey(a + b)) throw new InvalidOperationException($\"Merged token '{a}{b}' missing from vocab; regenerate or re-download model files\");","typeGuard":"null","tryCatchPattern":"try { var tok = new BpeTokenizer(vocab, merges); }\ncatch (InvalidOperationException ex) when (ex.Message.StartsWith(\"Trying to merge a token\")) { throw new InvalidDataException(\"Vocab is missing tokens produced by merge rules; use unpruned vocab.json\", ex); }","preventionTips":["Avoid frequency-pruned vocab files that drop tokens referenced by merges.","Keep vocab and merges from the same training checkpoint.","Run a construction smoke test in CI for each model artifact set."],"tags":["vocabulary","merge-file","bpe","model-files"],"backgroundTag":"entity-not-found","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}