dotnet/machinelearning · error · ArgumentException

The beginning of sentence token must be provided when the fl

Error message

The beginning of sentence token must be provided when the flag is set to include it in the encoding.

What it means

When the AddBeginningOfSentence flag is true, encoding must prepend the BOS token, so a null/empty BeginningOfSentenceToken makes the tokenizer unusable. The constructor throws ArgumentException to reject this inconsistent option combination up front.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs:181

                        throw new ArgumentException($"The beginning of sentence token '{BeginningOfSentenceToken}' is not found in the vocabulary.");
                    }

                    BeginningOfSentenceId = value.beggingOfSentenceId;
                }

                if (!string.IsNullOrEmpty(EndOfSentenceToken))
                {
                    if (!_vocab.TryGetValue(EndOfSentenceToken!, out (int endOfSentenceId, string token) value))
                    {
                        throw new ArgumentException($"The end of sentence token '{EndOfSentenceToken}' is not found in the vocabulary.");
                    }

                    EndOfSentenceId = value.endOfSentenceId;
                }

                if (AddBeginningOfSentence && string.IsNullOrEmpty(BeginningOfSentenceToken))
                {
                    throw new ArgumentException("The beginning of sentence token must be provided when the flag is set to include it in the encoding.");
                }

                if (AddEndOfSentence && string.IsNullOrEmpty(EndOfSentenceToken))
                {
                    throw new ArgumentException("The end of sentence token must be provided when the flag is set to include it in the encoding.");
                }
            }
            finally
            {
                if (disposeStream)
                {
                    vocabularyStream.Dispose();
                    mergeStream.Dispose();
                }
            }
        }

        /// <summary>

View on GitHub (pinned to 7b76e69cf9)

Solutions

Provide a valid BeginningOfSentenceToken that exists in the vocabulary when AddBeginningOfSentence is true.
Set AddBeginningOfSentence = false if you do not want BOS prepended.
Validate the options object for flag/token consistency before calling CodeGenTokenizer.Create.

Example fix

// before
var opts = new CodeGenOptions { AddBeginningOfSentence = true }; // no token
// after
var opts = new CodeGenOptions { AddBeginningOfSentence = true, BeginningOfSentenceToken = "<|endoftext|>" };

Defensive patterns

Strategy: validation

Validate before calling

if (opts.AddBeginningOfSentence && string.IsNullOrEmpty(opts.BeginningOfSentenceToken))
    throw new InvalidOperationException("AddBeginningOfSentence requires BeginningOfSentenceToken");

Type guard

null

Try / catch

try { var tok = CodeGenTokenizer.Create(vocabStream, opts); }
catch (ArgumentException ex) when (ex.Message.Contains("beginning of sentence token must be provided")) { /* supply token or disable flag */ }

Prevention

Build tokenizer options through a factory that enforces flag/token consistency
Validate the options object in unit tests for both BOS and EOS combinations
Default the token string when the include-flag is set

When it happens

Trigger: Creating CodeGenTokenizer with AddBeginningOfSentence = true while BeginningOfSentenceToken is null or the empty string in the options.

Common situations: Setting the include-BOS flag copied from sample code but forgetting to specify the token string; building options programmatically where the token variable was never assigned.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/5b7eb878a4781753. Report an issue: GitHub.