dotnet/machinelearning · error · ArgumentException

The end of sentence token must be provided when the flag is

Error message

The end of sentence token must be provided when the flag is set to include it in the encoding.

What it means

Mirror of the BOS consistency check: AddEndOfSentence = true requires a non-empty EndOfSentenceToken, otherwise the tokenizer could not honor the flag during EncodeToIds. The constructor throws ArgumentException for this invalid option combination.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs:186

                if (!string.IsNullOrEmpty(EndOfSentenceToken))
                {
                    if (!_vocab.TryGetValue(EndOfSentenceToken!, out (int endOfSentenceId, string token) value))
                    {
                        throw new ArgumentException($"The end of sentence token '{EndOfSentenceToken}' is not found in the vocabulary.");
                    }

                    EndOfSentenceId = value.endOfSentenceId;
                }

                if (AddBeginningOfSentence && string.IsNullOrEmpty(BeginningOfSentenceToken))
                {
                    throw new ArgumentException("The beginning of sentence token must be provided when the flag is set to include it in the encoding.");
                }

                if (AddEndOfSentence && string.IsNullOrEmpty(EndOfSentenceToken))
                {
                    throw new ArgumentException("The end of sentence token must be provided when the flag is set to include it in the encoding.");
                }
            }
            finally
            {
                if (disposeStream)
                {
                    vocabularyStream.Dispose();
                    mergeStream.Dispose();
                }
            }
        }

        /// <summary>
        /// Gets the added tokens.
        /// </summary>
        public IReadOnlyDictionary<string, int>? SpecialTokens { get; }

        /// <summary>

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Provide a valid EndOfSentenceToken that exists in the vocabulary when AddEndOfSentence is true.
2. Set AddEndOfSentence = false if EOS appending is unwanted.
3. Validate flag/token consistency before constructing the tokenizer.

Example fix

// before
var opts = new CodeGenOptions { AddEndOfSentence = true }; // no token
// after
var opts = new CodeGenOptions { AddEndOfSentence = true, EndOfSentenceToken = "<|endoftext|>" };

if (opts.AddEndOfSentence && string.IsNullOrEmpty(opts.EndOfSentenceToken))
    throw new InvalidOperationException("AddEndOfSentence requires EndOfSentenceToken");

null

try { var tok = CodeGenTokenizer.Create(vocabStream, opts); }
catch (ArgumentException ex) when (ex.Message.Contains("end of sentence token must be provided")) { /* supply token or disable flag */ }

When it happens

Trigger: Creating CodeGenTokenizer with AddEndOfSentence = true while EndOfSentenceToken is null or empty.

Common situations: Flag set from configuration or sample code with the token never specified; options built dynamically where the token assignment was skipped.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• The beginning of sentence token must be provided when the fl
• The beginning of sentence token '{BeginningOfSentenceToken}'
• The end of sentence token '{EndOfSentenceToken}' is not foun
• Parameter must not be null, empty, or whitespace
• String.Format(Strings.MismatchedColumnValueType, this.DataTy

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/68017b933052b027. Report an issue: GitHub.