dotnet/machinelearning · error · ArgumentException

The end of sentence token

Error message

The end of sentence token '{EndOfSentenceToken}' is not found in the vocabulary.

What it means

Symmetric to the BOS check: when EndOfSentenceToken is non-empty, the constructor verifies it exists as a key in the vocabulary dictionary and throws ArgumentException if not. The tokenizer needs the token's ID to append EOS during encoding, so an unknown token cannot be honored.

Solutions

1. Set EndOfSentenceToken to the exact special-token string present in the vocabulary (for CodeGen typically '<|endoftext|>').
2. Confirm membership by deserializing the vocab JSON and checking the token key before constructing the tokenizer.
3. Leave EndOfSentenceToken null/empty if EOS handling is not needed.
4. Use the vocabulary file shipped with the exact CodeGen checkpoint rather than a foreign one.

Example fix

// before
new CodeGenOptions { EndOfSentenceToken = "</s>" } // GPT-style token, absent from CodeGen vocab
// after
new CodeGenOptions { EndOfSentenceToken = "<|endoftext|>" }

var vocab = JsonSerializer.Deserialize<Dictionary<string,int>>(vocabJson);
if (!string.IsNullOrEmpty(opts.EndOfSentenceToken) && !vocab.ContainsKey(opts.EndOfSentenceToken))
    throw new InvalidOperationException($"EOS token '{opts.EndOfSentenceToken}' missing from vocabulary");

bool IsInVocab(string? token, Dictionary<string,int> vocab) => !string.IsNullOrEmpty(token) && vocab.ContainsKey(token);

try { var tok = CodeGenTokenizer.Create(vocabStream, opts); }
catch (ArgumentException ex) when (ex.Message.Contains("end of sentence token")) { /* fix options / fallback */ }

When it happens

Trigger: Constructing CodeGenTokenizer with options whose EndOfSentenceToken (e.g. '</s>' borrowed from Llama, or a typo like '<endofext>') is not a key in the supplied vocabulary JSON stream.

Common situations: Mixing special tokens across model families, editing vocab.json by hand, or pointing the tokenizer at a partial/outdated vocabulary export.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• The beginning of sentence token
• Failed to read the vocabulary file.
• Problems met when parsing JSON vocabulary object.
• The beginning of sentence token must be provided when the…
• The end of sentence token must be provided when the flag is…

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/31869b44f95f1d78. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs:173

                    UnknownTokenId = value.unknownId;
                }

                if (!string.IsNullOrEmpty(BeginningOfSentenceToken))
                {
                    if (!_vocab.TryGetValue(BeginningOfSentenceToken!, out (int beggingOfSentenceId, string token) value))
                    {
                        throw new ArgumentException($"The beginning of sentence token '{BeginningOfSentenceToken}' is not found in the vocabulary.");
                    }

                    BeginningOfSentenceId = value.beggingOfSentenceId;
                }

                if (!string.IsNullOrEmpty(EndOfSentenceToken))
                {
                    if (!_vocab.TryGetValue(EndOfSentenceToken!, out (int endOfSentenceId, string token) value))
                    {
                        throw new ArgumentException($"The end of sentence token '{EndOfSentenceToken}' is not found in the vocabulary.");
                    }

                    EndOfSentenceId = value.endOfSentenceId;
                }

                if (AddBeginningOfSentence && string.IsNullOrEmpty(BeginningOfSentenceToken))
                {
                    throw new ArgumentException("The beginning of sentence token must be provided when the flag is set to include it in the encoding.");
                }

                if (AddEndOfSentence && string.IsNullOrEmpty(EndOfSentenceToken))
                {
                    throw new ArgumentException("The end of sentence token must be provided when the flag is set to include it in the encoding.");
                }
            }
            finally
            {
                if (disposeStream)

View on GitHub (pinned to 7b76e69cf9)