dotnet/machinelearning · error · ArgumentException

The Unknown token '{UnknownToken}' is not found in the vocab

Error message

The Unknown token '{UnknownToken}' is not found in the vocabulary.

What it means

CodeGenTokenizer treats UnknownToken as a special token that must map to an id in the loaded vocabulary. During construction it looks up UnknownToken in _vocab and throws ArgumentException when the token string is not present. This guarantees the tokenizer can always fall back to a valid unknown-token id at encode time.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs:153

                {
                    SpecialTokens = specialTokens;
                    _specialTokens = specialTokens.ToDictionary(kvp => new StringSpanOrdinalKey(kvp.Key), kvp => (kvp.Value, kvp.Key));
                    _specialTokensReverse = specialTokens.ToDictionary(kvp => kvp.Value, kvp => kvp.Key);
                }

                UnknownToken = unknownToken;
                BeginningOfSentenceToken = beginningOfSentenceToken;
                EndOfSentenceToken = endOfSentenceToken;

                AddPrefixSpace = addPrefixSpace;
                AddBeginningOfSentence = addBeginningOfSentence;
                AddEndOfSentence = addEndOfSentence;

                if (!string.IsNullOrEmpty(UnknownToken))
                {
                    if (!_vocab.TryGetValue(UnknownToken!, out (int unknownId, string token) value))
                    {
                        throw new ArgumentException($"The Unknown token '{UnknownToken}' is not found in the vocabulary.");
                    }

                    UnknownTokenId = value.unknownId;
                }

                if (!string.IsNullOrEmpty(BeginningOfSentenceToken))
                {
                    if (!_vocab.TryGetValue(BeginningOfSentenceToken!, out (int beggingOfSentenceId, string token) value))
                    {
                        throw new ArgumentException($"The beginning of sentence token '{BeginningOfSentenceToken}' is not found in the vocabulary.");
                    }

                    BeginningOfSentenceId = value.beggingOfSentenceId;
                }

                if (!string.IsNullOrEmpty(EndOfSentenceToken))
                {
                    if (!_vocab.TryGetValue(EndOfSentenceToken!, out (int endOfSentenceId, string token) value))

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Pass an unknownToken string that exists verbatim in the vocabulary file (check vocab.json keys).
2. Omit the unknownToken argument to use the library default that matches the model.
3. If the model has no unknown token, pass an explicit empty/none value consistent with the vocab instead of guessing.

Example fix

// before
var tokenizer = new CodeGenTokenizer(vocabPath, mergesPath, unknownToken: "<unk>");
// after: verify the token exists first
var vocab = JsonSerializer.Deserialize<Dictionary<string, int>>(File.ReadAllText(vocabPath))!;
string unknown = vocab.ContainsKey("<unk>") ? "<unk>" : vocab.Keys.First(k => k.Contains("unk"));
var tokenizer = new CodeGenTokenizer(vocabPath, mergesPath, unknownToken: unknown);

var vocab = JsonSerializer.Deserialize<Dictionary<string, int>>(File.ReadAllText(vocabPath))!;
if (unknownToken is not null && unknownToken.Length > 0 && !vocab.ContainsKey(unknownToken))
    throw new ArgumentException($"unknownToken '{unknownToken}' not in vocabulary");

try { var tok = new CodeGenTokenizer(vocabPath, mergesPath, unknownToken: unknownToken); } catch (ArgumentException ex) when (ex.Message.Contains("Unknown token")) { logger.LogError(ex, "Unknown token {Tok} missing from vocab", unknownToken); }

When it happens

Trigger: Constructing CodeGenTokenizer with unknownToken set to a string absent from vocab.json, e.g. unknownToken: "<unk>" while the CodeGen vocab uses "<|unknown|>" or no unknown token at all.

Common situations: Copying default special-token values between tokenizer models with different vocab conventions; overriding unknownToken while loading a vocab from a different model release.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• The special token '{kvp.Key}' is not in the vocabulary or as
• The special token '{token}' is not in the vocabulary.
• throw new ArgumentNullException(nameof(vocabStream));
• The beginning of sentence token '{beginningOfSentenceToken}'
• The end of sentence token '{endOfSentenceToken}' was not pre

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/92a0f64e06d7a4e8. Report an issue: GitHub.