dotnet/machinelearning · error · NotSupportedException

tokenizer.json post_processor template does not contain a se

Error message

tokenizer.json post_processor template does not contain a sequence placeholder.

What it means

A TemplateProcessing template must include exactly one Sequence placeholder marking where the encoded input goes. If ProcessTemplate finishes without seeing any 'Sequence' item, the post-processor could only ever emit special tokens and never the input, so NotSupportedException is thrown.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs:912

                    seenSequence = true;
                }
                else if (item.TryGetProperty("SpecialToken", out JsonElement specialToken) &&
                         specialToken.TryGetProperty("id", out JsonElement idElement))
                {
                    if (idElement.ValueKind != JsonValueKind.String)
                    {
                        throw new InvalidDataException("A post_processor template 'SpecialToken.id' must be a string.");
                    }

                    string tokenName = idElement.GetString()!;
                    int id = ResolveTemplateTokenId(tokenName, ppSpecialTokens, specialTokens, vocab);
                    (seenSequence ? suffixTokens : prefixTokens).Add((id, tokenName));
                }
            }

            if (!seenSequence)
            {
                throw new NotSupportedException("tokenizer.json post_processor template does not contain a sequence placeholder.");
            }
        }

        private static int ResolveTemplateTokenId(
            string tokenName,
            JsonElement? ppSpecialTokens,
            IReadOnlyDictionary<string, int> specialTokens,
            IReadOnlyList<(string Piece, float Score)> vocab)
        {
            if (ppSpecialTokens is JsonElement st &&
                st.TryGetProperty(tokenName, out JsonElement entry) &&
                entry.TryGetProperty("ids", out JsonElement ids) &&
                ids.ValueKind == JsonValueKind.Array &&
                ids.GetArrayLength() > 0)
            {
                if (ids[0].ValueKind != JsonValueKind.Number)
                {
                    throw new InvalidDataException($"The tokenizer.json post_processor special token '{tokenName}' has a non-numeric id.");

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Insert a {"Sequence": {}} item into the template where the input tokens should appear, e.g. ["<s>", {"Sequence": {}}, "</s>"]
2. Regenerate the post_processor via tokenizer.save with the original HuggingFace configuration
3. Remove the post_processor entirely and append special tokens manually after encoding
4. Pre-validate the template JSON contains one Sequence element before constructing the tokenizer

Example fix

// before (single template)
["<s>", "</s>"]
// after
["<s>", {"Sequence": {}}, "</s>"]

bool hasSeq = template.EnumerateArray().Any(i => i.TryGetProperty("Sequence", out _));
if (!hasSeq) throw new FormatException("Template missing Sequence placeholder");

static bool HasSequencePlaceholder(JsonElement template) =>
    template.ValueKind == JsonValueKind.Array &&
    template.EnumerateArray().Any(i => i.TryGetProperty("Sequence", out _));

try { tok = SentencePieceTokenizer.Create(...); }
catch (NotSupportedException ex) when (ex.Message.Contains("sequence placeholder"))
{ /* repair template or drop post_processor */ }

When it happens

Trigger: tokenizer.json post_processor template whose 'single' (or 'pair') list contains only SpecialToken items and no {"Sequence": {}} placeholder; templates copied from configs where the sequence was accidentally deleted.

Common situations: Hand-trimming a template and removing $A/{Sequence} by mistake; aggregating only BOS/EOS tokens and assuming the sequence is implicit; corrupted or partially written tokenizer.json.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• tokenizer.json post_processor templates with more than one s
• A post_processor template 'SpecialToken.id' must be a string
• The tokenizer.json post_processor special token '{tokenName}
• The tokenizer.json post_processor special token '{tokenName}
• The post-processor '{property}' token '{token}' with id {id}

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/209a6c42e20c7032. Report an issue: GitHub.