{"record":{"id":"09ad32e56e225698","repo":"dotnet/machinelearning","slug":"tokenizer-json-post-processor-templates-with-more","errorCode":null,"errorMessage":"tokenizer.json post_processor templates with more than one sequence are not supported.","messagePattern":"tokenizer\\.json post_processor templates with more than one sequence are not supported\\.","errorType":"validation","errorClass":"NotSupportedException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs","lineNumber":891,"sourceCode":"                return;\n            }\n\n            JsonElement? ppSpecialTokens = postProcessor.TryGetProperty(\"special_tokens\", out JsonElement st) && st.ValueKind == JsonValueKind.Object\n                ? st : (JsonElement?)null;\n\n            bool seenSequence = false;\n            foreach (JsonElement item in single.EnumerateArray())\n            {\n                if (item.ValueKind != JsonValueKind.Object)\n                {\n                    continue;\n                }\n\n                if (item.TryGetProperty(\"Sequence\", out _))\n                {\n                    if (seenSequence)\n                    {\n                        throw new NotSupportedException(\"tokenizer.json post_processor templates with more than one sequence are not supported.\");\n                    }\n\n                    seenSequence = true;\n                }\n                else if (item.TryGetProperty(\"SpecialToken\", out JsonElement specialToken) &&\n                         specialToken.TryGetProperty(\"id\", out JsonElement idElement))\n                {\n                    if (idElement.ValueKind != JsonValueKind.String)\n                    {\n                        throw new InvalidDataException(\"A post_processor template 'SpecialToken.id' must be a string.\");\n                    }\n\n                    string tokenName = idElement.GetString()!;\n                    int id = ResolveTemplateTokenId(tokenName, ppSpecialTokens, specialTokens, vocab);\n                    (seenSequence ? suffixTokens : prefixTokens).Add((id, tokenName));\n                }\n            }\n","sourceCodeStart":873,"sourceCodeEnd":909,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs#L873-L909","documentation":"ProcessTemplate walks a post_processor TemplateProcessing 'single'/'pair' template and only supports at most one Sequence placeholder (the input sequence). Templates containing a second 'Sequence' item cannot be represented by the single prefix/suffix special-token affix model, so NotSupportedException is thrown at load time.","triggerScenarios":"Constructing SentencePieceTokenizer from a tokenizer.json whose post_processor.type is TemplateProcessing and whose template contains two or more 'Sequence' placeholders (e.g. a pair template like \"<s> A </s> B </s>\" with two Sequences, or a duplicated Sequence in the single template).","commonSituations":"Reusing a BERT/XLM-R style tokenizer.json (built for pair encoding with two sequences) with SentencePiece; hand-merging post-processor configs from different models; copying a template that was valid in Python tokenizers but unsupported here.","solutions":["Use a tokenizer.json whose post_processor template contains exactly one Sequence placeholder plus SpecialToken items","Simplify the template to a single-sequence form, e.g. \"<s> $A </s>\" written as [SpecialToken, Sequence, SpecialToken]","Remove or disable the post_processor section in tokenizer.json and add the special-token affixes manually in code","Catch NotSupportedException and fall back to a tokenizer variant without the template post-processor"],"exampleFix":"// before (post_processor single template)\n[\"<s>\", {\"Sequence\": {}}, \"</s>\", {\"Sequence\": {}}]\n// after\n[\"<s>\", {\"Sequence\": {}}, \"</s>\"]","handlingStrategy":"validation","validationCode":"var pp = doc.RootElement.GetProperty(\"post_processor\");\nint seqCount = CountSequences(pp.GetProperty(\"single\"));\nif (seqCount > 1) throw new NotSupportedException(\"Template has multiple Sequence placeholders\");","typeGuard":"static bool IsSingleSequenceTemplate(JsonElement pp) =>\n    pp.TryGetProperty(\"single\", out var t) &&\n    t.EnumerateArray().Count(i => i.TryGetProperty(\"Sequence\", out _)) == 1;","tryCatchPattern":"try { tok = SentencePieceTokenizer.Create(...); }\ncatch (NotSupportedException ex) when (ex.Message.Contains(\"post_processor templates\"))\n{ tok = LoadWithoutPostProcessor(...); }","preventionTips":["Use SentencePiece tokenizer.json files, not BERT-style pair templates","Inspect the post_processor single/pair templates before loading","Regenerate the post_processor with HuggingFace tokenizers when in doubt"],"tags":["tokenizer","unsupported-feature","post-processor","template"],"backgroundTag":"unsupported-operation","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}