{"record":{"id":"136f784a69e65092","repo":"dotnet/machinelearning","slug":"a-post-processor-template-specialtoken-id-must-b","errorCode":null,"errorMessage":"A post_processor template 'SpecialToken.id' must be a string.","messagePattern":"A post_processor template 'SpecialToken\\.id' must be a string\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs","lineNumber":901,"sourceCode":"                {\n                    continue;\n                }\n\n                if (item.TryGetProperty(\"Sequence\", out _))\n                {\n                    if (seenSequence)\n                    {\n                        throw new NotSupportedException(\"tokenizer.json post_processor templates with more than one sequence are not supported.\");\n                    }\n\n                    seenSequence = true;\n                }\n                else if (item.TryGetProperty(\"SpecialToken\", out JsonElement specialToken) &&\n                         specialToken.TryGetProperty(\"id\", out JsonElement idElement))\n                {\n                    if (idElement.ValueKind != JsonValueKind.String)\n                    {\n                        throw new InvalidDataException(\"A post_processor template 'SpecialToken.id' must be a string.\");\n                    }\n\n                    string tokenName = idElement.GetString()!;\n                    int id = ResolveTemplateTokenId(tokenName, ppSpecialTokens, specialTokens, vocab);\n                    (seenSequence ? suffixTokens : prefixTokens).Add((id, tokenName));\n                }\n            }\n\n            if (!seenSequence)\n            {\n                throw new NotSupportedException(\"tokenizer.json post_processor template does not contain a sequence placeholder.\");\n            }\n        }\n\n        private static int ResolveTemplateTokenId(\n            string tokenName,\n            JsonElement? ppSpecialTokens,\n            IReadOnlyDictionary<string, int> specialTokens,","sourceCodeStart":883,"sourceCodeEnd":919,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs#L883-L919","documentation":"In a TemplateProcessing post_processor template, a SpecialToken item's 'id' must be the token's string text (e.g. \"</s>\"). If the JSON 'id' property exists but is not a string (a number, object, etc.), ProcessTemplate throws InvalidDataException because it cannot resolve the token name against the vocabulary.","triggerScenarios":"Loading tokenizer.json where a post_processor template SpecialToken entry has a numeric or non-string 'id' (e.g. {\"SpecialToken\": {\"id\": 2, \"ids\": [2]}} instead of \"id\": \"</s>\"); hand-written templates confusing the string token name with the numeric vocab id.","commonSituations":"Authors confusing SpecialToken.id (string token text) with the numeric id array; converting post-processor configs between formats; manual edits after partially reading HuggingFace docs.","solutions":["Change SpecialToken.id in the template to the token's string text, e.g. {\"SpecialToken\": {\"id\": \"</s>\", \"ids\": [2], \"type_id\": 0}}","Keep the numeric value in the companion 'ids' array (ids[0] must be a number) while 'id' stays a string","Regenerate the post_processor section with HuggingFace tokenizers save instead of hand-editing","Pre-validate that every SpecialToken.id in single/pair templates is a JSON string before loading"],"exampleFix":"// before\n{\"SpecialToken\": {\"id\": 2, \"ids\": [2]}}\n// after\n{\"SpecialToken\": {\"id\": \"</s>\", \"ids\": [2]}}","handlingStrategy":"validation","validationCode":"foreach (var item in template.EnumerateArray())\n    if (item.TryGetProperty(\"SpecialToken\", out var st) && st.TryGetProperty(\"id\", out var id)\n        && id.ValueKind != JsonValueKind.String) throw new FormatException(\"SpecialToken.id must be a string\");","typeGuard":"static bool HasStringSpecialTokenId(JsonElement item) =>\n    !item.TryGetProperty(\"SpecialToken\", out var st) ||\n    !st.TryGetProperty(\"id\", out var id) || id.ValueKind == JsonValueKind.String;","tryCatchPattern":"try { tok = SentencePieceTokenizer.Create(...); }\ncatch (InvalidDataException ex) when (ex.Message.Contains(\"SpecialToken.id\"))\n{ /* fix template or fall back */ }","preventionTips":["Remember SpecialToken.id is the token TEXT; numbers go in 'ids'","Copy template structure from HuggingFace docs exactly","Prefer regenerated tokenizer.json over manual edits"],"tags":["json","tokenizer","post-processor","type-mismatch"],"backgroundTag":"type-mismatch","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}