{"record":{"id":"67e133b02ab7b80a","repo":"dotnet/machinelearning","slug":"the-special-token-token-is-not-in-the-vocabula","errorCode":null,"errorMessage":"The special token '{token}' is not in the vocabulary.","messagePattern":"The special token '(.+?)' is not in the vocabulary\\.","errorType":"exception","errorClass":"ArgumentException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/BertTokenizer.cs","lineNumber":818,"sourceCode":"                    AddSpecialToken(vocab, tempSpecialTokens, options.MaskingToken, lowerCase, notNormalizedSpecialTokens);\n\n                    options.SpecialTokens = notNormalizedSpecialTokens;\n                    specialTokensDict = tempSpecialTokens;\n                }\n            }\n\n            // We set the PreTokenizer here using the normalized special tokens dict (if relevant), and therefore we can \n            // keep the not-normalized special tokens dict in the options passed to the WordPieceTokenizer.\n            options.PreTokenizer ??= options.ApplyBasicTokenization ? PreTokenizer.CreateWordOrPunctuation(options.SplitOnSpecialTokens ? specialTokensDict : null) : PreTokenizer.CreateWhiteSpace();\n\n            return new BertTokenizer(vocab, vocabReverse, options);\n        }\n\n        private static void AddSpecialToken(Dictionary<StringSpanOrdinalKey, int> vocab, Dictionary<string, int> specialTokens, string token, bool lowerCase, Dictionary<string, int>? notNormalizedSpecialTokens = null)\n        {\n            if (token is null || !vocab.TryGetValue(new StringSpanOrdinalKey(token), out int id))\n            {\n                throw new ArgumentException($\"The special token '{token}' is not in the vocabulary.\");\n            }\n\n            if (notNormalizedSpecialTokens is not null)\n            {\n                notNormalizedSpecialTokens[token] = id;\n            }\n\n            string normalizedToken = token;\n            if (lowerCase)\n            {\n                // Lowercase the special tokens to have the pre-tokenization can find them as we lowercase the input text.\n                // we don't even need to do case-insensitive comparisons as we are lowercasing the input text.\n                normalizedToken = token.ToLowerInvariant();\n\n                // Add lowercased special tokens to the vocab if they are not already there.\n                // This will allow matching during the encoding process.\n                vocab[new StringSpanOrdinalKey(normalizedToken)] = id;\n            }","sourceCodeStart":800,"sourceCodeEnd":836,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/BertTokenizer.cs#L800-L836","documentation":"The private helper AddSpecialToken throws ArgumentException when the token is null or is not present in the vocabulary. It is invoked from Create while registering options.SpecialTokens, so a token absent from vocab.txt cannot be registered as special. Matching against the vocab is ordinal, so casing/spelling must be exact.","triggerScenarios":"A special token string in BertOptions.SpecialTokens that does not appear in the vocabulary, or a null token entry; reached via BertTokenizer.Create with mismatched options rather than called directly by user code.","commonSituations":"Vocabulary files missing standard tokens like [CLS]/[SEP]/[MASK] (truncated or custom-built vocabs); token strings with casing/whitespace that don't match vocab entries; IDs corrected to match but the token string itself still misspelled.","solutions":["Use only token strings that literally exist in the vocabulary file.","Verify exact spelling/case of the special token against vocab.txt (matching is ordinal).","If the token is genuinely missing, extend the vocabulary or remove it from SpecialTokens.","Ensure no null entries are added to the SpecialTokens dictionary."],"exampleFix":"// before\noptions.SpecialTokens[\"[SEPERATOR]\"] = 102; // misspelled, not in vocab\n// after\noptions.SpecialTokens[\"[SEP]\"] = 102; // exact vocab entry","handlingStrategy":"validation","validationCode":"// Confirm each special token exists in vocab.txt before construction\nbool inVocab = File.ReadLines(vocabPath).Contains(token, StringComparer.Ordinal);","typeGuard":"static bool IsKnownSpecialToken(string? token, HashSet<string> vocab) => !string.IsNullOrEmpty(token) && vocab.Contains(token);","tryCatchPattern":"try { var t = BertTokenizer.Create(vocabPath, options); } catch (ArgumentException ex) when (ex.Message.Contains(\"is not in the vocabulary\")) { /* drop the unknown special token and retry */ }","preventionTips":["Use exact token strings from vocab.txt (ordinal matching, case-sensitive).","Validate custom vocabs include [PAD]/[UNK]/[CLS]/[SEP]/[MASK] before use.","Avoid hand-typing special token names; copy from the vocab file."],"tags":["csharp","invalid-argument","vocabulary","tokenizer"],"backgroundTag":"invalid-argument-value","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}