{"record":{"id":"9e1366817dea5bcf","repo":"dotnet/machinelearning","slug":"the-beginning-of-sentence-token-beginningofsente","errorCode":null,"errorMessage":"The beginning of sentence token '{beginningOfSentenceToken}' was not present in the vocabulary.","messagePattern":"The beginning of sentence token '(.+?)' was not present in the vocabulary\\.","errorType":"exception","errorClass":"InvalidOperationException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs","lineNumber":325,"sourceCode":"                    bool fuseUnknownTokens,\n                    bool byteLevel = false,\n                    string? beginningOfSentenceToken = null,\n                    string? endOfSentenceToken = null)\n        {\n            FuseUnknownTokens = fuseUnknownTokens;\n            ContinuingSubwordPrefix = continuingSubwordPrefix;\n            EndOfWordSuffix = endOfWordSuffix;\n            ByteLevel = byteLevel;\n            _preTokenizer = preTokenizer ?? PreTokenizer.CreateWordOrNonWord(); // Default to WordOrNonWord pre-tokenizer\n            _normalizer = normalizer;\n\n            _vocab = vocab ?? new Dictionary<StringSpanOrdinalKey, int>();\n\n            if (beginningOfSentenceToken is not null)\n            {\n                if (_vocab.TryGetValue(beginningOfSentenceToken, out int aId) is false && specialTokens?.TryGetValue(beginningOfSentenceToken, out aId) is false)\n                {\n                    throw new InvalidOperationException($\"The beginning of sentence token '{beginningOfSentenceToken}' was not present in the vocabulary.\");\n                }\n\n                BeginningOfSentenceId = aId;\n                BeginningOfSentenceToken = beginningOfSentenceToken;\n            }\n\n            if (endOfSentenceToken is not null)\n            {\n                if (_vocab.TryGetValue(endOfSentenceToken, out int aId) is false && specialTokens?.TryGetValue(endOfSentenceToken, out aId) is false)\n                {\n                    throw new InvalidOperationException($\"The end of sentence token '{endOfSentenceToken}' was not present in the vocabulary.\");\n                }\n\n                EndOfSentenceId = aId;\n                EndOfSentenceToken = endOfSentenceToken;\n            }\n\n            Cache = new StringSpanOrdinalKeyCache<Word>();","sourceCodeStart":307,"sourceCodeEnd":343,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs#L307-L343","documentation":"The BpeTokenizer constructor verifies that a configured beginning-of-sentence token exists either in the model vocabulary or in the special-tokens map; if found in neither, it throws InvalidOperationException. A BOS token that is not in the vocab cannot be encoded, so the tokenizer would be internally inconsistent if construction continued.","triggerScenarios":"new BpeTokenizer(vocab, merges, ..., beginningOfSentenceToken: \"<s>\", ...) where the vocab dictionary and specialTokens dictionary contain no \"<s>\" key — typically a mismatch between the BOS token string and the actual token names in the model files.","commonSituations":"Copying BOS token strings (e.g. \"<s>\" vs \"[CLS]\" vs \"<|begin_of_text|>\") between different models; hardcoding BOS tokens for models like Llama/DeepSeek whose vocab uses different markers; loading a truncated or partial vocab.json that dropped special tokens.","solutions":["Use the exact BOS token string from the model's vocab.json (or tokenizer config) as beginningOfSentenceToken.","Add the BOS token to the specialTokens dictionary passed to the constructor if it should be special but absent from vocab.","Pass beginningOfSentenceToken: null if you do not need BOS handling.","Verify the vocab file is complete and not truncated."],"exampleFix":"// before\nvar tok = new BpeTokenizer(vocab, merges, specialTokens: null, beginningOfSentenceToken: \"<s>\"); // '<s>' not in vocab\n// after\nvar specialTokens = new Dictionary<string, int> { [\"<s>\"] = 1, [\"</s>\"] = 2 };\nvar tok = new BpeTokenizer(vocab, merges, specialTokens: specialTokens, beginningOfSentenceToken: \"<s>\");","handlingStrategy":"validation","validationCode":"if (bos is not null && !vocab.ContainsKey(bos) && (specialTokens is null || !specialTokens.ContainsKey(bos)))\n    throw new InvalidOperationException($\"BOS token '{bos}' is not in vocab or special tokens\");","typeGuard":"static bool TokenExists(string? tok, Dictionary<string,int>? special) => tok is null || special?.ContainsKey(tok) == true;","tryCatchPattern":"try { var tok = new BpeTokenizer(vocab, merges, specialTokens, beginningOfSentenceToken: bos); }\ncatch (InvalidOperationException ex) when (ex.Message.Contains(\"beginning of sentence\")) { bos = null; /* retry without BOS */ }","preventionTips":["Copy BOS/EOS token strings from the model's tokenizer config files, never from memory.","Assert at startup that configured special tokens exist in the loaded vocab.","Keep special tokens in the specialTokens map when the vocab omits them."],"tags":["invalid-state","vocabulary","bpe","tokenizer"],"backgroundTag":"entity-not-found","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}