{"record":{"id":"d4dff125415829b7","repo":"dotnet/machinelearning","slug":"the-special-token-kvp-key-is-not-in-the-vocabu","errorCode":null,"errorMessage":"The special token '{kvp.Key}' is not in the vocabulary or assigned id value {id} different than the value {kvp.Value} in the special tokens.","messagePattern":"The special token '(.+?)' is not in the vocabulary or assigned id value (.+?) different than the value (.+?) in the special tokens\\.","errorType":"exception","errorClass":"ArgumentException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/BertTokenizer.cs","lineNumber":780,"sourceCode":"\n            options.Normalizer ??= options.ApplyBasicTokenization ? new BertNormalizer(options.LowerCaseBeforeTokenization, options.IndividuallyTokenizeCjk, options.RemoveNonSpacingMarks) : null;\n\n            IReadOnlyDictionary<string, int>? specialTokensDict = options.SpecialTokens;\n            if (options.SplitOnSpecialTokens)\n            {\n                bool lowerCase = options.ApplyBasicTokenization && options.LowerCaseBeforeTokenization;\n                if (options.SpecialTokens is not null)\n                {\n                    if (lowerCase)\n                    {\n                        Dictionary<string, int> tempSpecialTokens = [];\n                        specialTokensDict = tempSpecialTokens;\n\n                        foreach (var kvp in options.SpecialTokens)\n                        {\n                            if (!vocab.TryGetValue(new StringSpanOrdinalKey(kvp.Key), out int id) || id != kvp.Value)\n                            {\n                                throw new ArgumentException($\"The special token '{kvp.Key}' is not in the vocabulary or assigned id value {id} different than the value {kvp.Value} in the special tokens.\");\n                            }\n\n                            // Add the special token into our dictionary, normalizing it, and adding it into the\n                            // main vocab, if needed. \n                            AddSpecialToken(vocab, tempSpecialTokens, kvp.Key, true);\n                        }\n                    }\n                }\n                else\n                {\n                    // Create a dictionary with the special tokens - store the un-normalized forms in the options as\n                    // that field is exposed to the public. In addition, store the normalized form for creating the \n                    // pre-tokenizer.\n                    Dictionary<string, int> tempSpecialTokens = [];\n                    Dictionary<string, int> notNormalizedSpecialTokens = [];\n                    AddSpecialToken(vocab, tempSpecialTokens, options.UnknownToken, lowerCase, notNormalizedSpecialTokens);\n                    AddSpecialToken(vocab, tempSpecialTokens, options.SeparatorToken, lowerCase, notNormalizedSpecialTokens);\n                    AddSpecialToken(vocab, tempSpecialTokens, options.PaddingToken, lowerCase, notNormalizedSpecialTokens);","sourceCodeStart":762,"sourceCodeEnd":798,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/BertTokenizer.cs#L762-L798","documentation":"During BertTokenizer construction, every entry in BertOptions.SpecialTokens must already exist in the vocabulary with the exact same ID. If a special token string is not found in the vocab, or its vocab ID differs from the ID given in the options, Create throws ArgumentException. This keeps the tokenizer's internal special-token bookkeeping consistent with the vocabulary.","triggerScenarios":"Passing BertOptions.SpecialTokens containing a token string absent from the vocab file; or mapping a token to a different integer ID than the vocab assigns (e.g. copying IDs from a different model's config).","commonSituations":"Hand-copying special token maps (CLS=101, SEP=102, etc.) from another BERT variant whose vocab IDs differ; typos in special token strings; mixing a vocab.txt from one model with special tokens defined for another.","solutions":["Remove or correct the mismatched entry in BertOptions.SpecialTokens so token and ID match the vocab file.","Open the vocab file and confirm the token's actual ID, then use that ID in options.","If the token genuinely doesn't exist, drop it from SpecialTokens or use a vocab that contains it.","Load special tokens programmatically from the same source as the vocab rather than hardcoding."],"exampleFix":"// before\nvar options = new BertOptions { SpecialTokens = { [\"[CLS]\"] = 99 } }; // wrong id\nvar tokenizer = BertTokenizer.Create(vocabPath, options);\n// after\nvar options = new BertOptions { SpecialTokens = { [\"[CLS]\"] = 101 } }; // id as assigned in vocab.txt\nvar tokenizer = BertTokenizer.Create(vocabPath, options);","handlingStrategy":"validation","validationCode":"// Verify special tokens against the vocab before creating options\nforeach (var kvp in options.SpecialTokens)\n{\n    if (!vocabLookup.TryGetValue(kvp.Key, out var id) || id != kvp.Value)\n        throw new InvalidOperationException($\"Special token '{kvp.Key}' (id {kvp.Value}) does not match vocab\");\n}","typeGuard":null,"tryCatchPattern":"try { var t = BertTokenizer.Create(vocabPath, options); } catch (ArgumentException ex) when (ex.Message.Contains(\"special token\")) { /* retry with corrected/empty SpecialTokens */ }","preventionTips":["Read special token IDs from the same model config that produced the vocab.","Never hardcode IDs copied from a different BERT variant.","Log and validate every SpecialTokens entry against vocab.txt during deployment."],"tags":["csharp","invalid-argument","vocabulary","tokenizer"],"backgroundTag":"invalid-argument-value","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}