{"record":{"id":"e8e287241e54d814","repo":"dotnet/machinelearning","slug":"the-beginning-of-sentence-token-beginningofsente-e8e287","errorCode":null,"errorMessage":"The beginning of sentence token '{BeginningOfSentenceToken}' is not found in the vocabulary.","messagePattern":"The beginning of sentence token '(.+?)' is not found in the vocabulary\\.","errorType":"exception","errorClass":"ArgumentException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs","lineNumber":163,"sourceCode":"                AddPrefixSpace = addPrefixSpace;\n                AddBeginningOfSentence = addBeginningOfSentence;\n                AddEndOfSentence = addEndOfSentence;\n\n                if (!string.IsNullOrEmpty(UnknownToken))\n                {\n                    if (!_vocab.TryGetValue(UnknownToken!, out (int unknownId, string token) value))\n                    {\n                        throw new ArgumentException($\"The Unknown token '{UnknownToken}' is not found in the vocabulary.\");\n                    }\n\n                    UnknownTokenId = value.unknownId;\n                }\n\n                if (!string.IsNullOrEmpty(BeginningOfSentenceToken))\n                {\n                    if (!_vocab.TryGetValue(BeginningOfSentenceToken!, out (int beggingOfSentenceId, string token) value))\n                    {\n                        throw new ArgumentException($\"The beginning of sentence token '{BeginningOfSentenceToken}' is not found in the vocabulary.\");\n                    }\n\n                    BeginningOfSentenceId = value.beggingOfSentenceId;\n                }\n\n                if (!string.IsNullOrEmpty(EndOfSentenceToken))\n                {\n                    if (!_vocab.TryGetValue(EndOfSentenceToken!, out (int endOfSentenceId, string token) value))\n                    {\n                        throw new ArgumentException($\"The end of sentence token '{EndOfSentenceToken}' is not found in the vocabulary.\");\n                    }\n\n                    EndOfSentenceId = value.endOfSentenceId;\n                }\n\n                if (AddBeginningOfSentence && string.IsNullOrEmpty(BeginningOfSentenceToken))\n                {\n                    throw new ArgumentException(\"The beginning of sentence token must be provided when the flag is set to include it in the encoding.\");","sourceCodeStart":145,"sourceCodeEnd":181,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs#L145-L181","documentation":"CodeGenTokenizer's constructor validates that the BeginningOfSentenceToken string exists as a key in the deserialized vocabulary dictionary. If the token is non-empty but not present in _vocab, it throws ArgumentException because BOS encoding would be impossible. This is a fail-fast constructor check that keeps the tokenizer in a usable state.","triggerScenarios":"Creating a CodeGenTokenizer (e.g. CodeGenTokenizer.Create) with options whose BeginningOfSentenceToken (e.g. '<|endoftext|>' misspelled, or a token from a different model like '</s>') is not a key in the supplied vocabulary JSON stream.","commonSituations":"Using BOS/EOS tokens copied from another model family (GPT-2 vs CodeGen), a hand-edited vocab.json, or a truncated vocabulary file that lacks the special token.","solutions":["Fix the BeginningOfSentenceToken option to the exact token string present in the vocabulary (for CodeGen models typically '<|endoftext|>').","Verify the token exists by loading the vocab JSON and checking that it contains the token key before constructing the tokenizer.","Leave BeginningOfSentenceToken null/empty if you do not need BOS handling, so the check is skipped.","Regenerate/redownload the vocabulary file for the exact model checkpoint you are using."],"exampleFix":"// before\nvar tok = CodeGenTokenizer.Create(vocabStream, new CodeGenOptions { BeginningOfSentenceToken = \"<s>\" }); // not in vocab\n// after\nvar tok = CodeGenTokenizer.Create(vocabStream, new CodeGenOptions { BeginningOfSentenceToken = \"<|endoftext|>\" }); // present in vocab.json","handlingStrategy":"validation","validationCode":"var vocab = JsonSerializer.Deserialize<Dictionary<string,int>>(vocabJson);\nif (!string.IsNullOrEmpty(opts.BeginningOfSentenceToken) && !vocab.ContainsKey(opts.BeginningOfSentenceToken))\n    throw new InvalidOperationException($\"BOS token '{opts.BeginningOfSentenceToken}' missing from vocabulary\");","typeGuard":"bool IsInVocab(string? token, Dictionary<string,int> vocab) => !string.IsNullOrEmpty(token) && vocab.ContainsKey(token);","tryCatchPattern":"try { var tok = CodeGenTokenizer.Create(vocabStream, opts); }\ncatch (ArgumentException ex) when (ex.Message.Contains(\"beginning of sentence token\")) { /* fix options / fallback to default tokenizer */ }","preventionTips":["Keep special tokens in a shared constant per model family instead of typing them inline","Validate BOS/EOS tokens against the vocab before constructing the tokenizer","Never copy special-token strings between different model families"],"tags":["tokenizer","vocabulary","argument-validation","dotnet"],"backgroundTag":"resource-not-found","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}