{"record":{"id":"5b7eb878a4781753","repo":"dotnet/machinelearning","slug":"the-beginning-of-sentence-token-must-be-provided-w","errorCode":null,"errorMessage":"The beginning of sentence token must be provided when the flag is set to include it in the encoding.","messagePattern":"The beginning of sentence token must be provided when the flag is set to include it in the encoding\\.","errorType":"exception","errorClass":"ArgumentException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs","lineNumber":181,"sourceCode":"                        throw new ArgumentException($\"The beginning of sentence token '{BeginningOfSentenceToken}' is not found in the vocabulary.\");\n                    }\n\n                    BeginningOfSentenceId = value.beggingOfSentenceId;\n                }\n\n                if (!string.IsNullOrEmpty(EndOfSentenceToken))\n                {\n                    if (!_vocab.TryGetValue(EndOfSentenceToken!, out (int endOfSentenceId, string token) value))\n                    {\n                        throw new ArgumentException($\"The end of sentence token '{EndOfSentenceToken}' is not found in the vocabulary.\");\n                    }\n\n                    EndOfSentenceId = value.endOfSentenceId;\n                }\n\n                if (AddBeginningOfSentence && string.IsNullOrEmpty(BeginningOfSentenceToken))\n                {\n                    throw new ArgumentException(\"The beginning of sentence token must be provided when the flag is set to include it in the encoding.\");\n                }\n\n                if (AddEndOfSentence && string.IsNullOrEmpty(EndOfSentenceToken))\n                {\n                    throw new ArgumentException(\"The end of sentence token must be provided when the flag is set to include it in the encoding.\");\n                }\n            }\n            finally\n            {\n                if (disposeStream)\n                {\n                    vocabularyStream.Dispose();\n                    mergeStream.Dispose();\n                }\n            }\n        }\n\n        /// <summary>","sourceCodeStart":163,"sourceCodeEnd":199,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs#L163-L199","documentation":"When the AddBeginningOfSentence flag is true, encoding must prepend the BOS token, so a null/empty BeginningOfSentenceToken makes the tokenizer unusable. The constructor throws ArgumentException to reject this inconsistent option combination up front.","triggerScenarios":"Creating CodeGenTokenizer with AddBeginningOfSentence = true while BeginningOfSentenceToken is null or the empty string in the options.","commonSituations":"Setting the include-BOS flag copied from sample code but forgetting to specify the token string; building options programmatically where the token variable was never assigned.","solutions":["Provide a valid BeginningOfSentenceToken that exists in the vocabulary when AddBeginningOfSentence is true.","Set AddBeginningOfSentence = false if you do not want BOS prepended.","Validate the options object for flag/token consistency before calling CodeGenTokenizer.Create."],"exampleFix":"// before\nvar opts = new CodeGenOptions { AddBeginningOfSentence = true }; // no token\n// after\nvar opts = new CodeGenOptions { AddBeginningOfSentence = true, BeginningOfSentenceToken = \"<|endoftext|>\" };","handlingStrategy":"validation","validationCode":"if (opts.AddBeginningOfSentence && string.IsNullOrEmpty(opts.BeginningOfSentenceToken))\n    throw new InvalidOperationException(\"AddBeginningOfSentence requires BeginningOfSentenceToken\");","typeGuard":"null","tryCatchPattern":"try { var tok = CodeGenTokenizer.Create(vocabStream, opts); }\ncatch (ArgumentException ex) when (ex.Message.Contains(\"beginning of sentence token must be provided\")) { /* supply token or disable flag */ }","preventionTips":["Build tokenizer options through a factory that enforces flag/token consistency","Validate the options object in unit tests for both BOS and EOS combinations","Default the token string when the include-flag is set"],"tags":["tokenizer","missing-token","argument-validation","dotnet"],"backgroundTag":"missing-required-argument","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}