{"record":{"id":"31869b44f95f1d78","repo":"dotnet/machinelearning","slug":"the-end-of-sentence-token-endofsentencetoken-i","errorCode":null,"errorMessage":"The end of sentence token '{EndOfSentenceToken}' is not found in the vocabulary.","messagePattern":"The end of sentence token '(.+?)' is not found in the vocabulary\\.","errorType":"exception","errorClass":"ArgumentException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs","lineNumber":173,"sourceCode":"\n                    UnknownTokenId = value.unknownId;\n                }\n\n                if (!string.IsNullOrEmpty(BeginningOfSentenceToken))\n                {\n                    if (!_vocab.TryGetValue(BeginningOfSentenceToken!, out (int beggingOfSentenceId, string token) value))\n                    {\n                        throw new ArgumentException($\"The beginning of sentence token '{BeginningOfSentenceToken}' is not found in the vocabulary.\");\n                    }\n\n                    BeginningOfSentenceId = value.beggingOfSentenceId;\n                }\n\n                if (!string.IsNullOrEmpty(EndOfSentenceToken))\n                {\n                    if (!_vocab.TryGetValue(EndOfSentenceToken!, out (int endOfSentenceId, string token) value))\n                    {\n                        throw new ArgumentException($\"The end of sentence token '{EndOfSentenceToken}' is not found in the vocabulary.\");\n                    }\n\n                    EndOfSentenceId = value.endOfSentenceId;\n                }\n\n                if (AddBeginningOfSentence && string.IsNullOrEmpty(BeginningOfSentenceToken))\n                {\n                    throw new ArgumentException(\"The beginning of sentence token must be provided when the flag is set to include it in the encoding.\");\n                }\n\n                if (AddEndOfSentence && string.IsNullOrEmpty(EndOfSentenceToken))\n                {\n                    throw new ArgumentException(\"The end of sentence token must be provided when the flag is set to include it in the encoding.\");\n                }\n            }\n            finally\n            {\n                if (disposeStream)","sourceCodeStart":155,"sourceCodeEnd":191,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs#L155-L191","documentation":"Symmetric to the BOS check: when EndOfSentenceToken is non-empty, the constructor verifies it exists as a key in the vocabulary dictionary and throws ArgumentException if not. The tokenizer needs the token's ID to append EOS during encoding, so an unknown token cannot be honored.","triggerScenarios":"Constructing CodeGenTokenizer with options whose EndOfSentenceToken (e.g. '</s>' borrowed from Llama, or a typo like '<endofext>') is not a key in the supplied vocabulary JSON stream.","commonSituations":"Mixing special tokens across model families, editing vocab.json by hand, or pointing the tokenizer at a partial/outdated vocabulary export.","solutions":["Set EndOfSentenceToken to the exact special-token string present in the vocabulary (for CodeGen typically '<|endoftext|>').","Confirm membership by deserializing the vocab JSON and checking the token key before constructing the tokenizer.","Leave EndOfSentenceToken null/empty if EOS handling is not needed.","Use the vocabulary file shipped with the exact CodeGen checkpoint rather than a foreign one."],"exampleFix":"// before\nnew CodeGenOptions { EndOfSentenceToken = \"</s>\" } // GPT-style token, absent from CodeGen vocab\n// after\nnew CodeGenOptions { EndOfSentenceToken = \"<|endoftext|>\" }","handlingStrategy":"validation","validationCode":"var vocab = JsonSerializer.Deserialize<Dictionary<string,int>>(vocabJson);\nif (!string.IsNullOrEmpty(opts.EndOfSentenceToken) && !vocab.ContainsKey(opts.EndOfSentenceToken))\n    throw new InvalidOperationException($\"EOS token '{opts.EndOfSentenceToken}' missing from vocabulary\");","typeGuard":"bool IsInVocab(string? token, Dictionary<string,int> vocab) => !string.IsNullOrEmpty(token) && vocab.ContainsKey(token);","tryCatchPattern":"try { var tok = CodeGenTokenizer.Create(vocabStream, opts); }\ncatch (ArgumentException ex) when (ex.Message.Contains(\"end of sentence token\")) { /* fix options / fallback */ }","preventionTips":["Define EOS constants per model in a static class","Cross-check EOS/BOS tokens against vocab.json during configuration load","Use the vocab file shipped with the exact checkpoint"],"tags":["tokenizer","vocabulary","argument-validation","dotnet"],"backgroundTag":"resource-not-found","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}