{"record":{"id":"f047374710685f6a","repo":"dotnet/machinelearning","slug":"the-end-of-sentence-token-endofsentencetoken-w","errorCode":null,"errorMessage":"The end of sentence token '{endOfSentenceToken}' was not present in the vocabulary.","messagePattern":"The end of sentence token '(.+?)' was not present in the vocabulary\\.","errorType":"exception","errorClass":"InvalidOperationException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs","lineNumber":336,"sourceCode":"\n            _vocab = vocab ?? new Dictionary<StringSpanOrdinalKey, int>();\n\n            if (beginningOfSentenceToken is not null)\n            {\n                if (_vocab.TryGetValue(beginningOfSentenceToken, out int aId) is false && specialTokens?.TryGetValue(beginningOfSentenceToken, out aId) is false)\n                {\n                    throw new InvalidOperationException($\"The beginning of sentence token '{beginningOfSentenceToken}' was not present in the vocabulary.\");\n                }\n\n                BeginningOfSentenceId = aId;\n                BeginningOfSentenceToken = beginningOfSentenceToken;\n            }\n\n            if (endOfSentenceToken is not null)\n            {\n                if (_vocab.TryGetValue(endOfSentenceToken, out int aId) is false && specialTokens?.TryGetValue(endOfSentenceToken, out aId) is false)\n                {\n                    throw new InvalidOperationException($\"The end of sentence token '{endOfSentenceToken}' was not present in the vocabulary.\");\n                }\n\n                EndOfSentenceId = aId;\n                EndOfSentenceToken = endOfSentenceToken;\n            }\n\n            Cache = new StringSpanOrdinalKeyCache<Word>();\n\n            VocabReverse = new();\n\n            foreach (KeyValuePair<StringSpanOrdinalKey, int> kvp in _vocab)\n            {\n                VocabReverse.Add(kvp.Value, kvp.Key.Data!);\n            }\n\n            if (specialTokens is not null)\n            {\n                SpecialTokens = specialTokens;","sourceCodeStart":318,"sourceCodeEnd":354,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs#L318-L354","documentation":"Analogous to the BOS check, the BpeTokenizer constructor requires the configured end-of-sentence token to exist in the vocabulary or the special-tokens map, and throws InvalidOperationException otherwise. Without a resolvable EOS id the tokenizer could not represent the end-of-sentence marker.","triggerScenarios":"new BpeTokenizer(vocab, merges, ..., endOfSentenceToken: \"</s>\", ...) where neither vocab nor specialTokens contains \"</s>\" — a string mismatch with the actual model files.","commonSituations":"Mixing EOS conventions across model families (\"</s>\" vs \"<|end_of_text|>\" vs \"[SEP]\"); loading a trimmed vocab; copy-pasting constructor args from another model's setup code.","solutions":["Use the exact EOS token string present in the model's vocab.json.","Add the EOS token to the specialTokens dictionary if it is missing from vocab.","Pass endOfSentenceToken: null when EOS handling is not needed.","Cross-check tokenizer_config.json / special_tokens_map.json for the correct token strings."],"exampleFix":"// before\nvar tok = new BpeTokenizer(vocab, merges, specialTokens: null, endOfSentenceToken: \"</s>\"); // missing\n// after\nvar specialTokens = new Dictionary<string, int> { [\"<|end_of_text|>\"] = 128001 };\nvar tok = new BpeTokenizer(vocab, merges, specialTokens: specialTokens, endOfSentenceToken: \"<|end_of_text|>\");","handlingStrategy":"validation","validationCode":"if (eos is not null && !vocab.ContainsKey(eos) && (specialTokens is null || !specialTokens.ContainsKey(eos)))\n    throw new InvalidOperationException($\"EOS token '{eos}' is not in vocab or special tokens\");","typeGuard":"static bool TokenExists(string? tok, Dictionary<string,int>? special) => tok is null || special?.ContainsKey(tok) == true;","tryCatchPattern":"try { var tok = new BpeTokenizer(vocab, merges, specialTokens, endOfSentenceToken: eos); }\ncatch (InvalidOperationException ex) when (ex.Message.Contains(\"end of sentence\")) { eos = null; /* retry without EOS */ }","preventionTips":["Source EOS token strings from the model's special_tokens_map.json.","Unit-test tokenizer construction against each supported model's real vocab.","Pass null for EOS when your scenario never uses end-of-sentence marking."],"tags":["invalid-state","vocabulary","bpe","tokenizer"],"backgroundTag":"entity-not-found","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}