{"record":{"id":"df4f4864a74444c1","repo":"dotnet/machinelearning","slug":"unknown-token-out-of-vocabulary","errorCode":null,"errorMessage":"Unknown Token Out Of Vocabulary.","messagePattern":"Unknown Token Out Of Vocabulary\\.","errorType":"exception","errorClass":"InvalidOperationException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs","lineNumber":1278,"sourceCode":"                    }\n                    word.Add(id, length);\n                }\n                else if (UnknownToken is not null)\n                {\n                    if (unk.HasValue)\n                    {\n                        if (FuseUnknownTokens)\n                        {\n                            // Fuse unk\n                            unk = (unk.Value.Id, unk.Value.Len + length);\n                        }\n                        else\n                        {\n                            // Do not fuse unk, add the previous one\n                            word.Add(unk.Value.Id, unk.Value.Len);\n                            if (!_vocab.TryGetValue(UnknownToken, out int value))\n                            {\n                                throw new InvalidOperationException($\"Unknown Token Out Of Vocabulary.\");\n                            }\n                            unk = (value, length);\n                        }\n                    }\n                    else\n                    {\n                        if (!_vocab.TryGetValue(UnknownToken, out int value))\n                        {\n                            throw new InvalidOperationException($\"Unknown Token Out Of Vocabulary.\");\n                        }\n                        unk = (value, length);\n                    }\n                }\n\n                i += length;\n            }\n\n            if (unk.HasValue)","sourceCodeStart":1260,"sourceCodeEnd":1296,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs#L1260-L1296","documentation":"During BPE encoding, when an unknown character sequence is fused and the tokenizer maps the UnknownToken (e.g. '<unk>') to its ID, that token must exist in the loaded vocabulary. If the vocab lacks an entry for UnknownToken, encoding fails with InvalidOperationException 'Unknown Token Out Of Vocabulary.'","triggerScenarios":"Encoding text containing characters absent from the vocabulary while the configured UnknownToken string has no entry in vocab.json — a mismatch between the UnknownToken setting and the actual vocab contents.","commonSituations":"Constructing BpeTokenizer with a custom unknownToken value that differs from the vocab's key (case/spacing mismatch), or using a vocab file with no <unk> entry and byte-level encoding disabled.","solutions":["Set UnknownToken to the exact string used in the vocabulary file (commonly '<unk>' or '<|unk|>').","Use a vocab that actually contains an unknown-token entry.","Prefer byte-level BPE tokenizers, whose byte fallback makes unknown tokens unnecessary."],"exampleFix":"// before\nvar opts = new BpeOptions(vocab) { UnknownToken = \"UNK\" }; // vocab has \"<unk>\", not \"UNK\"\n// after\nvar opts = new BpeOptions(vocab) { UnknownToken = \"<unk>\" };","handlingStrategy":"validation","validationCode":"if (!vocab.ContainsKey(unknownToken)) throw new InvalidOperationException($\"Vocab is missing unknown token '{unknownToken}'\");","typeGuard":null,"tryCatchPattern":"try { ids = tokenizer.EncodeToIds(text); }\ncatch (InvalidOperationException ex) when (ex.Message.Contains(\"Unknown Token Out Of Vocabulary\")) { /* fix vocab/options */ }","preventionTips":["Confirm the UnknownToken string matches the vocab key byte-for-byte.","Keep vocab and merges from the same model export.","Prefer byte-level BPE tokenizers that do not need an <unk> entry."],"tags":["csharp","tokenizers","vocabulary"],"backgroundTag":"internal-invariant-violation","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}