{"record":{"id":"68017b933052b027","repo":"dotnet/machinelearning","slug":"the-end-of-sentence-token-must-be-provided-when-th","errorCode":null,"errorMessage":"The end of sentence token must be provided when the flag is set to include it in the encoding.","messagePattern":"The end of sentence token must be provided when the flag is set to include it in the encoding\\.","errorType":"exception","errorClass":"ArgumentException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs","lineNumber":186,"sourceCode":"\n                if (!string.IsNullOrEmpty(EndOfSentenceToken))\n                {\n                    if (!_vocab.TryGetValue(EndOfSentenceToken!, out (int endOfSentenceId, string token) value))\n                    {\n                        throw new ArgumentException($\"The end of sentence token '{EndOfSentenceToken}' is not found in the vocabulary.\");\n                    }\n\n                    EndOfSentenceId = value.endOfSentenceId;\n                }\n\n                if (AddBeginningOfSentence && string.IsNullOrEmpty(BeginningOfSentenceToken))\n                {\n                    throw new ArgumentException(\"The beginning of sentence token must be provided when the flag is set to include it in the encoding.\");\n                }\n\n                if (AddEndOfSentence && string.IsNullOrEmpty(EndOfSentenceToken))\n                {\n                    throw new ArgumentException(\"The end of sentence token must be provided when the flag is set to include it in the encoding.\");\n                }\n            }\n            finally\n            {\n                if (disposeStream)\n                {\n                    vocabularyStream.Dispose();\n                    mergeStream.Dispose();\n                }\n            }\n        }\n\n        /// <summary>\n        /// Gets the added tokens.\n        /// </summary>\n        public IReadOnlyDictionary<string, int>? SpecialTokens { get; }\n\n        /// <summary>","sourceCodeStart":168,"sourceCodeEnd":204,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs#L168-L204","documentation":"Mirror of the BOS consistency check: AddEndOfSentence = true requires a non-empty EndOfSentenceToken, otherwise the tokenizer could not honor the flag during EncodeToIds. The constructor throws ArgumentException for this invalid option combination.","triggerScenarios":"Creating CodeGenTokenizer with AddEndOfSentence = true while EndOfSentenceToken is null or empty.","commonSituations":"Flag set from configuration or sample code with the token never specified; options built dynamically where the token assignment was skipped.","solutions":["Provide a valid EndOfSentenceToken that exists in the vocabulary when AddEndOfSentence is true.","Set AddEndOfSentence = false if EOS appending is unwanted.","Validate flag/token consistency before constructing the tokenizer."],"exampleFix":"// before\nvar opts = new CodeGenOptions { AddEndOfSentence = true }; // no token\n// after\nvar opts = new CodeGenOptions { AddEndOfSentence = true, EndOfSentenceToken = \"<|endoftext|>\" };","handlingStrategy":"validation","validationCode":"if (opts.AddEndOfSentence && string.IsNullOrEmpty(opts.EndOfSentenceToken))\n    throw new InvalidOperationException(\"AddEndOfSentence requires EndOfSentenceToken\");","typeGuard":"null","tryCatchPattern":"try { var tok = CodeGenTokenizer.Create(vocabStream, opts); }\ncatch (ArgumentException ex) when (ex.Message.Contains(\"end of sentence token must be provided\")) { /* supply token or disable flag */ }","preventionTips":["Enforce token/flag consistency at options-construction time","Add tests covering every flag combination","Set both BOS and EOS tokens together from model metadata"],"tags":["tokenizer","missing-token","argument-validation","dotnet"],"backgroundTag":"missing-required-argument","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}