dotnet/machinelearning · error · InvalidOperationException

The end of sentence token '{endOfSentenceToken}' was not pre

Error message

The end of sentence token '{endOfSentenceToken}' was not present in the vocabulary.

What it means

Analogous to the BOS check, the BpeTokenizer constructor requires the configured end-of-sentence token to exist in the vocabulary or the special-tokens map, and throws InvalidOperationException otherwise. Without a resolvable EOS id the tokenizer could not represent the end-of-sentence marker.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs:336

            _vocab = vocab ?? new Dictionary<StringSpanOrdinalKey, int>();

            if (beginningOfSentenceToken is not null)
            {
                if (_vocab.TryGetValue(beginningOfSentenceToken, out int aId) is false && specialTokens?.TryGetValue(beginningOfSentenceToken, out aId) is false)
                {
                    throw new InvalidOperationException($"The beginning of sentence token '{beginningOfSentenceToken}' was not present in the vocabulary.");
                }

                BeginningOfSentenceId = aId;
                BeginningOfSentenceToken = beginningOfSentenceToken;
            }

            if (endOfSentenceToken is not null)
            {
                if (_vocab.TryGetValue(endOfSentenceToken, out int aId) is false && specialTokens?.TryGetValue(endOfSentenceToken, out aId) is false)
                {
                    throw new InvalidOperationException($"The end of sentence token '{endOfSentenceToken}' was not present in the vocabulary.");
                }

                EndOfSentenceId = aId;
                EndOfSentenceToken = endOfSentenceToken;
            }

            Cache = new StringSpanOrdinalKeyCache<Word>();

            VocabReverse = new();

            foreach (KeyValuePair<StringSpanOrdinalKey, int> kvp in _vocab)
            {
                VocabReverse.Add(kvp.Value, kvp.Key.Data!);
            }

            if (specialTokens is not null)
            {
                SpecialTokens = specialTokens;

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Use the exact EOS token string present in the model's vocab.json.
2. Add the EOS token to the specialTokens dictionary if it is missing from vocab.
3. Pass endOfSentenceToken: null when EOS handling is not needed.
4. Cross-check tokenizer_config.json / special_tokens_map.json for the correct token strings.

Example fix

// before
var tok = new BpeTokenizer(vocab, merges, specialTokens: null, endOfSentenceToken: "</s>"); // missing
// after
var specialTokens = new Dictionary<string, int> { ["<|end_of_text|>"] = 128001 };
var tok = new BpeTokenizer(vocab, merges, specialTokens: specialTokens, endOfSentenceToken: "<|end_of_text|>");

if (eos is not null && !vocab.ContainsKey(eos) && (specialTokens is null || !specialTokens.ContainsKey(eos)))
    throw new InvalidOperationException($"EOS token '{eos}' is not in vocab or special tokens");

static bool TokenExists(string? tok, Dictionary<string,int>? special) => tok is null || special?.ContainsKey(tok) == true;

try { var tok = new BpeTokenizer(vocab, merges, specialTokens, endOfSentenceToken: eos); }
catch (InvalidOperationException ex) when (ex.Message.Contains("end of sentence")) { eos = null; /* retry without EOS */ }

When it happens

Trigger: new BpeTokenizer(vocab, merges, ..., endOfSentenceToken: "</s>", ...) where neither vocab nor specialTokens contains "</s>" — a string mismatch with the actual model files.

Common situations: Mixing EOS conventions across model families ("</s>" vs "<|end_of_text|>" vs "[SEP]"); loading a trimmed vocab; copy-pasting constructor args from another model's setup code.

Understand the failure class

Background: "Not found" and "does not exist" errors: why "Task not found", "No such folder", and "Can't find" fire when a lookup comes back empty — this error's family across 14 libraries.

Related errors

• The beginning of sentence token '{beginningOfSentenceToken}'
• Unknown Token '{value}' was not present in '{nameof(Vocabula
• The vocabulary cannot be null.
• throw new ArgumentNullException(nameof(vocabStream));
• Trying to merge a token '{mergeValues.a}' which not exist in

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/f047374710685f6a. Report an issue: GitHub.