dotnet/machinelearning · error · NotSupportedException

tokenizer.json post_processor templates with more than one s

Error message

tokenizer.json post_processor templates with more than one sequence are not supported.

What it means

ProcessTemplate walks a post_processor TemplateProcessing 'single'/'pair' template and only supports at most one Sequence placeholder (the input sequence). Templates containing a second 'Sequence' item cannot be represented by the single prefix/suffix special-token affix model, so NotSupportedException is thrown at load time.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs:891

                return;
            }

            JsonElement? ppSpecialTokens = postProcessor.TryGetProperty("special_tokens", out JsonElement st) && st.ValueKind == JsonValueKind.Object
                ? st : (JsonElement?)null;

            bool seenSequence = false;
            foreach (JsonElement item in single.EnumerateArray())
            {
                if (item.ValueKind != JsonValueKind.Object)
                {
                    continue;
                }

                if (item.TryGetProperty("Sequence", out _))
                {
                    if (seenSequence)
                    {
                        throw new NotSupportedException("tokenizer.json post_processor templates with more than one sequence are not supported.");
                    }

                    seenSequence = true;
                }
                else if (item.TryGetProperty("SpecialToken", out JsonElement specialToken) &&
                         specialToken.TryGetProperty("id", out JsonElement idElement))
                {
                    if (idElement.ValueKind != JsonValueKind.String)
                    {
                        throw new InvalidDataException("A post_processor template 'SpecialToken.id' must be a string.");
                    }

                    string tokenName = idElement.GetString()!;
                    int id = ResolveTemplateTokenId(tokenName, ppSpecialTokens, specialTokens, vocab);
                    (seenSequence ? suffixTokens : prefixTokens).Add((id, tokenName));
                }
            }

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Use a tokenizer.json whose post_processor template contains exactly one Sequence placeholder plus SpecialToken items
2. Simplify the template to a single-sequence form, e.g. "<s> $A </s>" written as [SpecialToken, Sequence, SpecialToken]
3. Remove or disable the post_processor section in tokenizer.json and add the special-token affixes manually in code
4. Catch NotSupportedException and fall back to a tokenizer variant without the template post-processor

Example fix

// before (post_processor single template)
["<s>", {"Sequence": {}}, "</s>", {"Sequence": {}}]
// after
["<s>", {"Sequence": {}}, "</s>"]

var pp = doc.RootElement.GetProperty("post_processor");
int seqCount = CountSequences(pp.GetProperty("single"));
if (seqCount > 1) throw new NotSupportedException("Template has multiple Sequence placeholders");

static bool IsSingleSequenceTemplate(JsonElement pp) =>
    pp.TryGetProperty("single", out var t) &&
    t.EnumerateArray().Count(i => i.TryGetProperty("Sequence", out _)) == 1;

try { tok = SentencePieceTokenizer.Create(...); }
catch (NotSupportedException ex) when (ex.Message.Contains("post_processor templates"))
{ tok = LoadWithoutPostProcessor(...); }

When it happens

Trigger: Constructing SentencePieceTokenizer from a tokenizer.json whose post_processor.type is TemplateProcessing and whose template contains two or more 'Sequence' placeholders (e.g. a pair template like "<s> A </s> B </s>" with two Sequences, or a duplicated Sequence in the single template).

Common situations: Reusing a BERT/XLM-R style tokenizer.json (built for pair encoding with two sequences) with SentencePiece; hand-merging post-processor configs from different models; copying a template that was valid in Python tokenizers but unsupported here.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• tokenizer.json post_processor template does not contain a se
• A post_processor template 'SpecialToken.id' must be a string
• The tokenizer.json post_processor special token '{tokenName}
• The tokenizer.json post_processor special token '{tokenName}
• The post-processor '{property}' token '{token}' with id {id}

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/09ad32e56e225698. Report an issue: GitHub.