{"record":{"id":"46803ff36581a88f","repo":"dotnet/machinelearning","slug":"the-tokenizer-json-post-processor-references-speci","errorCode":null,"errorMessage":"The tokenizer.json post_processor references special token '{tokenName}' that is not present in the vocabulary.","messagePattern":"The tokenizer\\.json post_processor references special token '(.+?)' that is not present in the vocabulary\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs","lineNumber":955,"sourceCode":"                bool consistent = (specialTokens.TryGetValue(tokenName, out int mappedId) && mappedId == id)\n                    || (id >= 0 && id < vocab.Count && vocab[id].Piece == tokenName);\n                if (!consistent)\n                {\n                    throw new InvalidDataException($\"The tokenizer.json post_processor special token '{tokenName}' maps to id {id}, which does not match the vocabulary or added tokens.\");\n                }\n\n                return id;\n            }\n\n            if (specialTokens.TryGetValue(tokenName, out int specialId))\n            {\n                return specialId;\n            }\n\n            int vocabId = FindPieceId(vocab, tokenName);\n            if (vocabId < 0)\n            {\n                throw new InvalidDataException($\"The tokenizer.json post_processor references special token '{tokenName}' that is not present in the vocabulary.\");\n            }\n\n            return vocabId;\n        }\n\n        private static void AddProcessorAffix(\n            JsonElement postProcessor,\n            string property,\n            List<(int Id, string Token)> target,\n            IReadOnlyList<(string Piece, float Score)> vocab,\n            IReadOnlyDictionary<string, int> specialTokens)\n        {\n            // Roberta/Bert processors store cls/sep as [token, id] arrays.\n            if (postProcessor.TryGetProperty(property, out JsonElement el) && el.ValueKind == JsonValueKind.Array && el.GetArrayLength() >= 2 &&\n                el[0].ValueKind == JsonValueKind.String && el[1].ValueKind == JsonValueKind.Number)\n            {\n                string token = el[0].GetString()!;\n                int id = el[1].GetInt32();","sourceCodeStart":937,"sourceCodeEnd":973,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs#L937-L973","documentation":"If a template references a special token name that appears in neither the post_processor special_tokens map, the added_tokens, nor the vocabulary, FindPieceId returns -1 and InvalidDataException is thrown. The tokenizer refuses to emit ids for tokens it cannot resolve.","triggerScenarios":"tokenizer.json post_processor template SpecialToken whose id string is not found via specialTokens or FindPieceId over the SentencePiece vocab — e.g. referencing \"[CLS]\" in a SentencePiece model that only defines \"<s>\", or a typo like \"<unk>\" vs \"unk\".","commonSituations":"Copying a post_processor (with BERT-style tokens) onto a SentencePiece tokenizer; renamed special tokens between tokenizer versions; typos in hand-edited templates.","solutions":["Correct the token name in the template to one present in the vocabulary or added_tokens (e.g. \"<s>\" instead of \"[CLS]\")","Add the missing token to added_tokens with a valid id, or add it to the vocab","Use a post_processor that belongs to the same tokenizer/model as the vocabulary","Pre-check each template token with a lookup over added_tokens and vocab before loading"],"exampleFix":"// before (SentencePiece vocab has no [CLS])\n{\"SpecialToken\": {\"id\": \"[CLS]\", \"ids\": [101]}}\n// after\n{\"SpecialToken\": {\"id\": \"<s>\", \"ids\": [1]}}","handlingStrategy":"validation","validationCode":"bool present = addedTokens.ContainsKey(name) || vocab.Contains(name);\nif (!present) throw new KeyNotFoundException($\"Special token '{name}' not in vocab\");","typeGuard":"static bool TokenExists(string name, IReadOnlyDictionary<string,int> added, IReadOnlyList<string> vocab) =>\n    added.ContainsKey(name) || vocab.Contains(name);","tryCatchPattern":"try { tok = SentencePieceTokenizer.Create(...); }\ncatch (InvalidDataException ex) when (ex.Message.Contains(\"not present in the vocabulary\"))\n{ /* add the token or use the matching post_processor */ }","preventionTips":["Match the post_processor to the tokenizer it shipped with","Check for SentencePiece (<s>) vs BERT ([CLS]) token naming differences","Spell-check template token names against the vocab"],"tags":["tokenizer","special-token","not-found","vocab"],"backgroundTag":"resource-not-found","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}