dotnet/machinelearning · error · InvalidDataException

A post_processor template 'SpecialToken.id' must be a string

Error message

A post_processor template 'SpecialToken.id' must be a string.

What it means

In a TemplateProcessing post_processor template, a SpecialToken item's 'id' must be the token's string text (e.g. "</s>"). If the JSON 'id' property exists but is not a string (a number, object, etc.), ProcessTemplate throws InvalidDataException because it cannot resolve the token name against the vocabulary.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs:901

                {
                    continue;
                }

                if (item.TryGetProperty("Sequence", out _))
                {
                    if (seenSequence)
                    {
                        throw new NotSupportedException("tokenizer.json post_processor templates with more than one sequence are not supported.");
                    }

                    seenSequence = true;
                }
                else if (item.TryGetProperty("SpecialToken", out JsonElement specialToken) &&
                         specialToken.TryGetProperty("id", out JsonElement idElement))
                {
                    if (idElement.ValueKind != JsonValueKind.String)
                    {
                        throw new InvalidDataException("A post_processor template 'SpecialToken.id' must be a string.");
                    }

                    string tokenName = idElement.GetString()!;
                    int id = ResolveTemplateTokenId(tokenName, ppSpecialTokens, specialTokens, vocab);
                    (seenSequence ? suffixTokens : prefixTokens).Add((id, tokenName));
                }
            }

            if (!seenSequence)
            {
                throw new NotSupportedException("tokenizer.json post_processor template does not contain a sequence placeholder.");
            }
        }

        private static int ResolveTemplateTokenId(
            string tokenName,
            JsonElement? ppSpecialTokens,
            IReadOnlyDictionary<string, int> specialTokens,

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Change SpecialToken.id in the template to the token's string text, e.g. {"SpecialToken": {"id": "</s>", "ids": [2], "type_id": 0}}
2. Keep the numeric value in the companion 'ids' array (ids[0] must be a number) while 'id' stays a string
3. Regenerate the post_processor section with HuggingFace tokenizers save instead of hand-editing
4. Pre-validate that every SpecialToken.id in single/pair templates is a JSON string before loading

Example fix

// before
{"SpecialToken": {"id": 2, "ids": [2]}}
// after
{"SpecialToken": {"id": "</s>", "ids": [2]}}

foreach (var item in template.EnumerateArray())
    if (item.TryGetProperty("SpecialToken", out var st) && st.TryGetProperty("id", out var id)
        && id.ValueKind != JsonValueKind.String) throw new FormatException("SpecialToken.id must be a string");

static bool HasStringSpecialTokenId(JsonElement item) =>
    !item.TryGetProperty("SpecialToken", out var st) ||
    !st.TryGetProperty("id", out var id) || id.ValueKind == JsonValueKind.String;

try { tok = SentencePieceTokenizer.Create(...); }
catch (InvalidDataException ex) when (ex.Message.Contains("SpecialToken.id"))
{ /* fix template or fall back */ }

When it happens

Trigger: Loading tokenizer.json where a post_processor template SpecialToken entry has a numeric or non-string 'id' (e.g. {"SpecialToken": {"id": 2, "ids": [2]}} instead of "id": "</s>"); hand-written templates confusing the string token name with the numeric vocab id.

Common situations: Authors confusing SpecialToken.id (string token text) with the numeric id array; converting post-processor configs between formats; manual edits after partially reading HuggingFace docs.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

• The tokenizer.json post_processor special token '{tokenName}
• Replace normalizer 'String' pattern must be a string.
• unknown option type
• The content of the vocabulary file '{vocabFile}' is not vali
• Problems met when parsing JSON vocabulary object.{Environmen

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/136f784a69e65092. Report an issue: GitHub.