{"record":{"id":"92a0f64e06d7a4e8","repo":"dotnet/machinelearning","slug":"the-unknown-token-unknowntoken-is-not-found-in","errorCode":null,"errorMessage":"The Unknown token '{UnknownToken}' is not found in the vocabulary.","messagePattern":"The Unknown token '(.+?)' is not found in the vocabulary\\.","errorType":"exception","errorClass":"ArgumentException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs","lineNumber":153,"sourceCode":"                {\n                    SpecialTokens = specialTokens;\n                    _specialTokens = specialTokens.ToDictionary(kvp => new StringSpanOrdinalKey(kvp.Key), kvp => (kvp.Value, kvp.Key));\n                    _specialTokensReverse = specialTokens.ToDictionary(kvp => kvp.Value, kvp => kvp.Key);\n                }\n\n                UnknownToken = unknownToken;\n                BeginningOfSentenceToken = beginningOfSentenceToken;\n                EndOfSentenceToken = endOfSentenceToken;\n\n                AddPrefixSpace = addPrefixSpace;\n                AddBeginningOfSentence = addBeginningOfSentence;\n                AddEndOfSentence = addEndOfSentence;\n\n                if (!string.IsNullOrEmpty(UnknownToken))\n                {\n                    if (!_vocab.TryGetValue(UnknownToken!, out (int unknownId, string token) value))\n                    {\n                        throw new ArgumentException($\"The Unknown token '{UnknownToken}' is not found in the vocabulary.\");\n                    }\n\n                    UnknownTokenId = value.unknownId;\n                }\n\n                if (!string.IsNullOrEmpty(BeginningOfSentenceToken))\n                {\n                    if (!_vocab.TryGetValue(BeginningOfSentenceToken!, out (int beggingOfSentenceId, string token) value))\n                    {\n                        throw new ArgumentException($\"The beginning of sentence token '{BeginningOfSentenceToken}' is not found in the vocabulary.\");\n                    }\n\n                    BeginningOfSentenceId = value.beggingOfSentenceId;\n                }\n\n                if (!string.IsNullOrEmpty(EndOfSentenceToken))\n                {\n                    if (!_vocab.TryGetValue(EndOfSentenceToken!, out (int endOfSentenceId, string token) value))","sourceCodeStart":135,"sourceCodeEnd":171,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs#L135-L171","documentation":"CodeGenTokenizer treats UnknownToken as a special token that must map to an id in the loaded vocabulary. During construction it looks up UnknownToken in _vocab and throws ArgumentException when the token string is not present. This guarantees the tokenizer can always fall back to a valid unknown-token id at encode time.","triggerScenarios":"Constructing CodeGenTokenizer with unknownToken set to a string absent from vocab.json, e.g. unknownToken: \"<unk>\" while the CodeGen vocab uses \"<|unknown|>\" or no unknown token at all.","commonSituations":"Copying default special-token values between tokenizer models with different vocab conventions; overriding unknownToken while loading a vocab from a different model release.","solutions":["Pass an unknownToken string that exists verbatim in the vocabulary file (check vocab.json keys).","Omit the unknownToken argument to use the library default that matches the model.","If the model has no unknown token, pass an explicit empty/none value consistent with the vocab instead of guessing."],"exampleFix":"// before\nvar tokenizer = new CodeGenTokenizer(vocabPath, mergesPath, unknownToken: \"<unk>\");\n// after: verify the token exists first\nvar vocab = JsonSerializer.Deserialize<Dictionary<string, int>>(File.ReadAllText(vocabPath))!;\nstring unknown = vocab.ContainsKey(\"<unk>\") ? \"<unk>\" : vocab.Keys.First(k => k.Contains(\"unk\"));\nvar tokenizer = new CodeGenTokenizer(vocabPath, mergesPath, unknownToken: unknown);","handlingStrategy":"validation","validationCode":"var vocab = JsonSerializer.Deserialize<Dictionary<string, int>>(File.ReadAllText(vocabPath))!;\nif (unknownToken is not null && unknownToken.Length > 0 && !vocab.ContainsKey(unknownToken))\n    throw new ArgumentException($\"unknownToken '{unknownToken}' not in vocabulary\");","typeGuard":null,"tryCatchPattern":"try { var tok = new CodeGenTokenizer(vocabPath, mergesPath, unknownToken: unknownToken); } catch (ArgumentException ex) when (ex.Message.Contains(\"Unknown token\")) { logger.LogError(ex, \"Unknown token {Tok} missing from vocab\", unknownToken); }","preventionTips":["Check vocab.json keys before overriding special tokens","Keep special-token defaults paired with the model they came from","Load the vocab once and validate all special tokens in a startup check"],"tags":["csharp","tokenizer","vocabulary"],"backgroundTag":"invalid-argument-value","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}