{"record":{"id":"027757019052c729","repo":"dotnet/machinelearning","slug":"the-tokenizer-json-post-processor-special-token-027757","errorCode":null,"errorMessage":"The tokenizer.json post_processor special token '{tokenName}' maps to id {id}, which does not match the vocabulary or added tokens.","messagePattern":"The tokenizer\\.json post_processor special token '(.+?)' maps to id (.+?), which does not match the vocabulary or added tokens\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs","lineNumber":941,"sourceCode":"                st.TryGetProperty(tokenName, out JsonElement entry) &&\n                entry.TryGetProperty(\"ids\", out JsonElement ids) &&\n                ids.ValueKind == JsonValueKind.Array &&\n                ids.GetArrayLength() > 0)\n            {\n                if (ids[0].ValueKind != JsonValueKind.Number)\n                {\n                    throw new InvalidDataException($\"The tokenizer.json post_processor special token '{tokenName}' has a non-numeric id.\");\n                }\n\n                int id = ids[0].GetInt32();\n\n                // Validate the id maps back to the referenced token (via added tokens or the vocab), mirroring\n                // AddProcessorAffix, so an inconsistent file cannot emit an id whose decoded token differs.\n                bool consistent = (specialTokens.TryGetValue(tokenName, out int mappedId) && mappedId == id)\n                    || (id >= 0 && id < vocab.Count && vocab[id].Piece == tokenName);\n                if (!consistent)\n                {\n                    throw new InvalidDataException($\"The tokenizer.json post_processor special token '{tokenName}' maps to id {id}, which does not match the vocabulary or added tokens.\");\n                }\n\n                return id;\n            }\n\n            if (specialTokens.TryGetValue(tokenName, out int specialId))\n            {\n                return specialId;\n            }\n\n            int vocabId = FindPieceId(vocab, tokenName);\n            if (vocabId < 0)\n            {\n                throw new InvalidDataException($\"The tokenizer.json post_processor references special token '{tokenName}' that is not present in the vocabulary.\");\n            }\n\n            return vocabId;\n        }","sourceCodeStart":923,"sourceCodeEnd":959,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs#L923-L959","documentation":"After resolving a template special token's numeric id, the library verifies consistency: the id must equal the id recorded in special_tokens, or vocab[id].Piece must equal the token text. An id that matches neither means decoding would yield a different token than intended, so InvalidDataException is thrown.","triggerScenarios":"tokenizer.json where post_processor special_tokens[tokenName].ids[0] disagrees with both the top-level added_tokens map and the vocabulary (e.g. ids shifted after vocab edits, or a copied special_tokens block from a different tokenizer).","commonSituations":"Merging a post_processor from one model with the vocab of another; editing the vocabulary after saving; mismatched tokenizer.json/vocab files fetched from different model revisions.","solutions":["Ensure special_tokens ids match either the added_tokens mapping or vocab[id].Piece == tokenName; regenerate the file from a single consistent tokenizer","Load vocab, added_tokens, and post_processor from the same model version — do not mix revisions","Re-export tokenizer.json with save_pretrained after any vocabulary change","Catch InvalidDataException and fall back to rebuilding affixes from added_tokens/vocab directly"],"exampleFix":"// before (vocab has \"</s>\" at 2 but special_tokens says ids: [5])\n\"</s>\": {\"ids\": [5]}\n// after\n\"</s>\": {\"ids\": [2]}","handlingStrategy":"validation","validationCode":"int id = specialTokens[token].ids[0];\nbool ok = (addedTokens.TryGetValue(token, out var m) && m == id) ||\n          (id >= 0 && id < vocab.Count && vocab[id] == token);\nif (!ok) throw new InvalidDataException(\"Inconsistent special token id\");","typeGuard":"static bool IsConsistentSpecialToken(string token, int id, IReadOnlyDictionary<string,int> added, IReadOnlyList<string> vocab) =>\n    (added.TryGetValue(token, out var m) && m == id) ||\n    (id >= 0 && id < vocab.Count && vocab[id] == token);","tryCatchPattern":"try { tok = SentencePieceTokenizer.Create(...); }\ncatch (InvalidDataException ex) when (ex.Message.Contains(\"does not match the vocabulary\"))\n{ /* reload all files from the same revision */ }","preventionTips":["Never mix vocab, added_tokens, and post_processor from different model revisions","Re-export the whole tokenizer after vocabulary changes","Pin tokenizer files by model revision hash"],"tags":["tokenizer","consistency","post-processor","invalid-data"],"backgroundTag":"invalid-argument-value","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}