{"record":{"id":"696c589365921242","repo":"dotnet/machinelearning","slug":"the-tokenizer-json-post-processor-special-token","errorCode":null,"errorMessage":"The tokenizer.json post_processor special token '{tokenName}' has a non-numeric id.","messagePattern":"The tokenizer\\.json post_processor special token '(.+?)' has a non-numeric id\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs","lineNumber":930,"sourceCode":"                throw new NotSupportedException(\"tokenizer.json post_processor template does not contain a sequence placeholder.\");\n            }\n        }\n\n        private static int ResolveTemplateTokenId(\n            string tokenName,\n            JsonElement? ppSpecialTokens,\n            IReadOnlyDictionary<string, int> specialTokens,\n            IReadOnlyList<(string Piece, float Score)> vocab)\n        {\n            if (ppSpecialTokens is JsonElement st &&\n                st.TryGetProperty(tokenName, out JsonElement entry) &&\n                entry.TryGetProperty(\"ids\", out JsonElement ids) &&\n                ids.ValueKind == JsonValueKind.Array &&\n                ids.GetArrayLength() > 0)\n            {\n                if (ids[0].ValueKind != JsonValueKind.Number)\n                {\n                    throw new InvalidDataException($\"The tokenizer.json post_processor special token '{tokenName}' has a non-numeric id.\");\n                }\n\n                int id = ids[0].GetInt32();\n\n                // Validate the id maps back to the referenced token (via added tokens or the vocab), mirroring\n                // AddProcessorAffix, so an inconsistent file cannot emit an id whose decoded token differs.\n                bool consistent = (specialTokens.TryGetValue(tokenName, out int mappedId) && mappedId == id)\n                    || (id >= 0 && id < vocab.Count && vocab[id].Piece == tokenName);\n                if (!consistent)\n                {\n                    throw new InvalidDataException($\"The tokenizer.json post_processor special token '{tokenName}' maps to id {id}, which does not match the vocabulary or added tokens.\");\n                }\n\n                return id;\n            }\n\n            if (specialTokens.TryGetValue(tokenName, out int specialId))\n            {","sourceCodeStart":912,"sourceCodeEnd":948,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs#L912-L948","documentation":"ResolveTemplateTokenId looks a template SpecialToken up in the post_processor's special_tokens map and reads ids[0] as its numeric vocabulary id. If that id exists but is not a JSON number, the id cannot be used to emit tokens, so InvalidDataException is thrown naming the offending token.","triggerScenarios":"tokenizer.json where post_processor.special_tokens[tokenName].ids[0] is a string, null, or other non-number (e.g. \"ids\": [\"2\"]); files edited so the ids array no longer matches the HuggingFace format [numericId].","commonSituations":"Serializers that stringify numbers; hand-edited special_tokens sections; cross-converted tokenizer configs from other tooling.","solutions":["Fix special_tokens so each entry's ids is an array of numbers, e.g. \"</s>\": {\"ids\": [2], \"type_id\": 0}","Regenerate tokenizer.json with HuggingFace tokenizers save_pretrained","Re-download the original tokenizer.json from the model hub","Pre-validate special_tokens entries: id property is a string and ids[0].ValueKind == Number"],"exampleFix":"// before\n\"</s>\": {\"id\": \"</s>\", \"ids\": [\"2\"]}\n// after\n\"</s>\": {\"id\": \"</s>\", \"ids\": [2]}","handlingStrategy":"validation","validationCode":"var ids = doc.RootElement.GetProperty(\"post_processor\")\n    .GetProperty(\"special_tokens\")[token].GetProperty(\"ids\");\nif (ids[0].ValueKind != JsonValueKind.Number) throw new FormatException(\"ids[0] must be numeric\");","typeGuard":"static bool HasNumericId(JsonElement specialToken) =>\n    specialToken.TryGetProperty(\"ids\", out var ids) && ids.ValueKind == JsonValueKind.Array &&\n    ids.GetArrayLength() > 0 && ids[0].ValueKind == JsonValueKind.Number;","tryCatchPattern":"try { tok = SentencePieceTokenizer.Create(...); }\ncatch (InvalidDataException ex) when (ex.Message.Contains(\"non-numeric id\"))\n{ /* fix special_tokens or re-export */ }","preventionTips":["Keep 'ids' arrays numeric per HuggingFace format","Avoid serializers that stringify numbers","Validate special_tokens shape before constructing the tokenizer"],"tags":["json","tokenizer","post-processor","type-mismatch"],"backgroundTag":"type-mismatch","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}