{"record":{"id":"209a6c42e20c7032","repo":"dotnet/machinelearning","slug":"tokenizer-json-post-processor-template-does-not-co","errorCode":null,"errorMessage":"tokenizer.json post_processor template does not contain a sequence placeholder.","messagePattern":"tokenizer\\.json post_processor template does not contain a sequence placeholder\\.","errorType":"validation","errorClass":"NotSupportedException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs","lineNumber":912,"sourceCode":"                    seenSequence = true;\n                }\n                else if (item.TryGetProperty(\"SpecialToken\", out JsonElement specialToken) &&\n                         specialToken.TryGetProperty(\"id\", out JsonElement idElement))\n                {\n                    if (idElement.ValueKind != JsonValueKind.String)\n                    {\n                        throw new InvalidDataException(\"A post_processor template 'SpecialToken.id' must be a string.\");\n                    }\n\n                    string tokenName = idElement.GetString()!;\n                    int id = ResolveTemplateTokenId(tokenName, ppSpecialTokens, specialTokens, vocab);\n                    (seenSequence ? suffixTokens : prefixTokens).Add((id, tokenName));\n                }\n            }\n\n            if (!seenSequence)\n            {\n                throw new NotSupportedException(\"tokenizer.json post_processor template does not contain a sequence placeholder.\");\n            }\n        }\n\n        private static int ResolveTemplateTokenId(\n            string tokenName,\n            JsonElement? ppSpecialTokens,\n            IReadOnlyDictionary<string, int> specialTokens,\n            IReadOnlyList<(string Piece, float Score)> vocab)\n        {\n            if (ppSpecialTokens is JsonElement st &&\n                st.TryGetProperty(tokenName, out JsonElement entry) &&\n                entry.TryGetProperty(\"ids\", out JsonElement ids) &&\n                ids.ValueKind == JsonValueKind.Array &&\n                ids.GetArrayLength() > 0)\n            {\n                if (ids[0].ValueKind != JsonValueKind.Number)\n                {\n                    throw new InvalidDataException($\"The tokenizer.json post_processor special token '{tokenName}' has a non-numeric id.\");","sourceCodeStart":894,"sourceCodeEnd":930,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs#L894-L930","documentation":"A TemplateProcessing template must include exactly one Sequence placeholder marking where the encoded input goes. If ProcessTemplate finishes without seeing any 'Sequence' item, the post-processor could only ever emit special tokens and never the input, so NotSupportedException is thrown.","triggerScenarios":"tokenizer.json post_processor template whose 'single' (or 'pair') list contains only SpecialToken items and no {\"Sequence\": {}} placeholder; templates copied from configs where the sequence was accidentally deleted.","commonSituations":"Hand-trimming a template and removing $A/{Sequence} by mistake; aggregating only BOS/EOS tokens and assuming the sequence is implicit; corrupted or partially written tokenizer.json.","solutions":["Insert a {\"Sequence\": {}} item into the template where the input tokens should appear, e.g. [\"<s>\", {\"Sequence\": {}}, \"</s>\"]","Regenerate the post_processor via tokenizer.save with the original HuggingFace configuration","Remove the post_processor entirely and append special tokens manually after encoding","Pre-validate the template JSON contains one Sequence element before constructing the tokenizer"],"exampleFix":"// before (single template)\n[\"<s>\", \"</s>\"]\n// after\n[\"<s>\", {\"Sequence\": {}}, \"</s>\"]","handlingStrategy":"validation","validationCode":"bool hasSeq = template.EnumerateArray().Any(i => i.TryGetProperty(\"Sequence\", out _));\nif (!hasSeq) throw new FormatException(\"Template missing Sequence placeholder\");","typeGuard":"static bool HasSequencePlaceholder(JsonElement template) =>\n    template.ValueKind == JsonValueKind.Array &&\n    template.EnumerateArray().Any(i => i.TryGetProperty(\"Sequence\", out _));","tryCatchPattern":"try { tok = SentencePieceTokenizer.Create(...); }\ncatch (NotSupportedException ex) when (ex.Message.Contains(\"sequence placeholder\"))\n{ /* repair template or drop post_processor */ }","preventionTips":["Always include exactly one {\"Sequence\": {}} in the template","Test tokenizer.json loading in CI on every config change","Keep a known-good copy of the tokenizer files"],"tags":["tokenizer","post-processor","template","unsupported-feature"],"backgroundTag":"schema-validation-failed","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}