{"record":{"id":"5adbe77e7f186578","repo":"dotnet/machinelearning","slug":"unknown-token-value-was-not-present-in-nameo","errorCode":null,"errorMessage":"Unknown Token '{value}' was not present in '{nameof(Vocabulary)}'.","messagePattern":"Unknown Token '(.+?)' was not present in '(.+?)'\\.","errorType":"exception","errorClass":"InvalidOperationException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs","lineNumber":62,"sourceCode":"        public string? UnknownToken\n        {\n            get\n            {\n                return _unknownToken;\n            }\n\n            private set\n            {\n                if (value is null)\n                {\n                    _unknownToken = value;\n                    _unknownTokenId = null;\n                    return;\n                }\n\n                if (!_vocab.TryGetValue(value, out int id))\n                {\n                    throw new InvalidOperationException($\"Unknown Token '{value}' was not present in '{nameof(Vocabulary)}'.\");\n                }\n\n                _unknownTokenId = id;\n                _unknownToken = value;\n            }\n        }\n\n        /// <summary>\n        /// A prefix to be used for every subword that is not a beginning-of-word\n        /// </summary>\n        public string? ContinuingSubwordPrefix { get; }\n\n        /// <summary>\n        /// An optional suffix to characterize the end-of-word and sub-word\n        /// </summary>\n        public string? EndOfWordSuffix { get; }\n\n        /// <summary>","sourceCodeStart":44,"sourceCodeEnd":80,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs#L44-L80","documentation":"The UnknownToken property setter of BpeTokenizer validates that the requested token actually exists in the loaded vocabulary; if the vocab lookup fails, it throws this InvalidOperationException. This fails fast so 'unknown token' handling does not silently map to a nonexistent id.","triggerScenarios":"Setting BpeTokenizer.UnknownToken (or BpeOptions.UnknownToken passed to BpeTokenizer.Create) to a string such as '<unk>' or '[UNK]' that is not a key in the supplied Vocabulary.","commonSituations":"Copying the unknown-token string from another tokenizer family (Hugging Face, SentencePiece) whose vocab uses a different sentinel; loading a trimmed/partial vocab file that dropped the <unk> entry; typos like '<unk>' vs '<UNK>'.","solutions":["Add the token you want as unknown (e.g. '<unk>') to the vocabulary file/dictionary with a valid id.","Use a token that is actually present in the vocabulary as UnknownToken (check the vocab file for the exact spelling).","Leave UnknownToken unset (null) if you do not need custom unknown-token handling — the setter returns early for null.","Verify with a vocab lookup before constructing the tokenizer."],"exampleFix":"// before\noptions.UnknownToken = \"<unk>\"; // not in vocab -> throws\n// after\noptions.Vocabulary[\"<unk>\"] = options.Vocabulary.Count; // add first\noptions.UnknownToken = \"<unk>\";","handlingStrategy":"validation","validationCode":"if (unknownToken != null && !vocab.ContainsKey(unknownToken))\n    throw new InvalidOperationException($\"UnknownToken '{unknownToken}' missing from vocabulary.\");","typeGuard":null,"tryCatchPattern":"try { options.UnknownToken = tok; tokenizer = BpeTokenizer.Create(options); }\ncatch (InvalidOperationException ex) when (ex.Message.Contains(\"was not present in\"))\n{ options.UnknownToken = null; tokenizer = BpeTokenizer.Create(options); }","preventionTips":["Confirm the exact unknown-token spelling against the vocab file before setting it.","Add the sentinel token to the vocab when building custom vocabularies.","Leave UnknownToken null unless custom unknown handling is required."],"tags":["dotnet","tokenizers","bpe","vocabulary"],"backgroundTag":"entity-not-found","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}