dotnet/machinelearning · error · ArgumentException

The beginning of sentence token '{BeginningOfSentenceToken}'

Error message

The beginning of sentence token '{BeginningOfSentenceToken}' is not found in the vocabulary.

What it means

CodeGenTokenizer's constructor validates that the BeginningOfSentenceToken string exists as a key in the deserialized vocabulary dictionary. If the token is non-empty but not present in _vocab, it throws ArgumentException because BOS encoding would be impossible. This is a fail-fast constructor check that keeps the tokenizer in a usable state.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs:163

                AddPrefixSpace = addPrefixSpace;
                AddBeginningOfSentence = addBeginningOfSentence;
                AddEndOfSentence = addEndOfSentence;

                if (!string.IsNullOrEmpty(UnknownToken))
                {
                    if (!_vocab.TryGetValue(UnknownToken!, out (int unknownId, string token) value))
                    {
                        throw new ArgumentException($"The Unknown token '{UnknownToken}' is not found in the vocabulary.");
                    }

                    UnknownTokenId = value.unknownId;
                }

                if (!string.IsNullOrEmpty(BeginningOfSentenceToken))
                {
                    if (!_vocab.TryGetValue(BeginningOfSentenceToken!, out (int beggingOfSentenceId, string token) value))
                    {
                        throw new ArgumentException($"The beginning of sentence token '{BeginningOfSentenceToken}' is not found in the vocabulary.");
                    }

                    BeginningOfSentenceId = value.beggingOfSentenceId;
                }

                if (!string.IsNullOrEmpty(EndOfSentenceToken))
                {
                    if (!_vocab.TryGetValue(EndOfSentenceToken!, out (int endOfSentenceId, string token) value))
                    {
                        throw new ArgumentException($"The end of sentence token '{EndOfSentenceToken}' is not found in the vocabulary.");
                    }

                    EndOfSentenceId = value.endOfSentenceId;
                }

                if (AddBeginningOfSentence && string.IsNullOrEmpty(BeginningOfSentenceToken))
                {
                    throw new ArgumentException("The beginning of sentence token must be provided when the flag is set to include it in the encoding.");

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Fix the BeginningOfSentenceToken option to the exact token string present in the vocabulary (for CodeGen models typically '<|endoftext|>').
2. Verify the token exists by loading the vocab JSON and checking that it contains the token key before constructing the tokenizer.
3. Leave BeginningOfSentenceToken null/empty if you do not need BOS handling, so the check is skipped.
4. Regenerate/redownload the vocabulary file for the exact model checkpoint you are using.

Example fix

// before
var tok = CodeGenTokenizer.Create(vocabStream, new CodeGenOptions { BeginningOfSentenceToken = "<s>" }); // not in vocab
// after
var tok = CodeGenTokenizer.Create(vocabStream, new CodeGenOptions { BeginningOfSentenceToken = "<|endoftext|>" }); // present in vocab.json

var vocab = JsonSerializer.Deserialize<Dictionary<string,int>>(vocabJson);
if (!string.IsNullOrEmpty(opts.BeginningOfSentenceToken) && !vocab.ContainsKey(opts.BeginningOfSentenceToken))
    throw new InvalidOperationException($"BOS token '{opts.BeginningOfSentenceToken}' missing from vocabulary");

bool IsInVocab(string? token, Dictionary<string,int> vocab) => !string.IsNullOrEmpty(token) && vocab.ContainsKey(token);

try { var tok = CodeGenTokenizer.Create(vocabStream, opts); }
catch (ArgumentException ex) when (ex.Message.Contains("beginning of sentence token")) { /* fix options / fallback to default tokenizer */ }

When it happens

Trigger: Creating a CodeGenTokenizer (e.g. CodeGenTokenizer.Create) with options whose BeginningOfSentenceToken (e.g. '<|endoftext|>' misspelled, or a token from a different model like '</s>') is not a key in the supplied vocabulary JSON stream.

Common situations: Using BOS/EOS tokens copied from another model family (GPT-2 vs CodeGen), a hand-edited vocab.json, or a truncated vocabulary file that lacks the special token.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• The end of sentence token '{EndOfSentenceToken}' is not foun
• The beginning of sentence token must be provided when the fl
• The end of sentence token must be provided when the flag is
• Problems met when parsing JSON vocabulary object.{Environmen
• Failed to read the vocabulary file.

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/e8e287241e54d814. Report an issue: GitHub.