dotnet/machinelearning · error · InvalidOperationException

The beginning of sentence token '{beginningOfSentenceToken}'

Error message

The beginning of sentence token '{beginningOfSentenceToken}' was not present in the vocabulary.

What it means

The BpeTokenizer constructor verifies that a configured beginning-of-sentence token exists either in the model vocabulary or in the special-tokens map; if found in neither, it throws InvalidOperationException. A BOS token that is not in the vocab cannot be encoded, so the tokenizer would be internally inconsistent if construction continued.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs:325

                    bool fuseUnknownTokens,
                    bool byteLevel = false,
                    string? beginningOfSentenceToken = null,
                    string? endOfSentenceToken = null)
        {
            FuseUnknownTokens = fuseUnknownTokens;
            ContinuingSubwordPrefix = continuingSubwordPrefix;
            EndOfWordSuffix = endOfWordSuffix;
            ByteLevel = byteLevel;
            _preTokenizer = preTokenizer ?? PreTokenizer.CreateWordOrNonWord(); // Default to WordOrNonWord pre-tokenizer
            _normalizer = normalizer;

            _vocab = vocab ?? new Dictionary<StringSpanOrdinalKey, int>();

            if (beginningOfSentenceToken is not null)
            {
                if (_vocab.TryGetValue(beginningOfSentenceToken, out int aId) is false && specialTokens?.TryGetValue(beginningOfSentenceToken, out aId) is false)
                {
                    throw new InvalidOperationException($"The beginning of sentence token '{beginningOfSentenceToken}' was not present in the vocabulary.");
                }

                BeginningOfSentenceId = aId;
                BeginningOfSentenceToken = beginningOfSentenceToken;
            }

            if (endOfSentenceToken is not null)
            {
                if (_vocab.TryGetValue(endOfSentenceToken, out int aId) is false && specialTokens?.TryGetValue(endOfSentenceToken, out aId) is false)
                {
                    throw new InvalidOperationException($"The end of sentence token '{endOfSentenceToken}' was not present in the vocabulary.");
                }

                EndOfSentenceId = aId;
                EndOfSentenceToken = endOfSentenceToken;
            }

            Cache = new StringSpanOrdinalKeyCache<Word>();

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Use the exact BOS token string from the model's vocab.json (or tokenizer config) as beginningOfSentenceToken.
2. Add the BOS token to the specialTokens dictionary passed to the constructor if it should be special but absent from vocab.
3. Pass beginningOfSentenceToken: null if you do not need BOS handling.
4. Verify the vocab file is complete and not truncated.

Example fix

// before
var tok = new BpeTokenizer(vocab, merges, specialTokens: null, beginningOfSentenceToken: "<s>"); // '<s>' not in vocab
// after
var specialTokens = new Dictionary<string, int> { ["<s>"] = 1, ["</s>"] = 2 };
var tok = new BpeTokenizer(vocab, merges, specialTokens: specialTokens, beginningOfSentenceToken: "<s>");

if (bos is not null && !vocab.ContainsKey(bos) && (specialTokens is null || !specialTokens.ContainsKey(bos)))
    throw new InvalidOperationException($"BOS token '{bos}' is not in vocab or special tokens");

static bool TokenExists(string? tok, Dictionary<string,int>? special) => tok is null || special?.ContainsKey(tok) == true;

try { var tok = new BpeTokenizer(vocab, merges, specialTokens, beginningOfSentenceToken: bos); }
catch (InvalidOperationException ex) when (ex.Message.Contains("beginning of sentence")) { bos = null; /* retry without BOS */ }

When it happens

Trigger: new BpeTokenizer(vocab, merges, ..., beginningOfSentenceToken: "<s>", ...) where the vocab dictionary and specialTokens dictionary contain no "<s>" key — typically a mismatch between the BOS token string and the actual token names in the model files.

Common situations: Copying BOS token strings (e.g. "<s>" vs "[CLS]" vs "<|begin_of_text|>") between different models; hardcoding BOS tokens for models like Llama/DeepSeek whose vocab uses different markers; loading a truncated or partial vocab.json that dropped special tokens.

Understand the failure class

Background: "Not found" and "does not exist" errors: why "Task not found", "No such folder", and "Can't find" fire when a lookup comes back empty — this error's family across 14 libraries.

Related errors

• The end of sentence token '{endOfSentenceToken}' was not pre
• Unknown Token '{value}' was not present in '{nameof(Vocabula
• The vocabulary cannot be null.
• throw new ArgumentNullException(nameof(vocabStream));
• Trying to merge a token '{mergeValues.a}' which not exist in

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/9e1366817dea5bcf. Report an issue: GitHub.