dotnet/machinelearning · error · InvalidDataException

The tokenizer.json post_processor special token '{tokenName}

Error message

The tokenizer.json post_processor special token '{tokenName}' has a non-numeric id.

What it means

ResolveTemplateTokenId looks a template SpecialToken up in the post_processor's special_tokens map and reads ids[0] as its numeric vocabulary id. If that id exists but is not a JSON number, the id cannot be used to emit tokens, so InvalidDataException is thrown naming the offending token.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs:930

                throw new NotSupportedException("tokenizer.json post_processor template does not contain a sequence placeholder.");
            }
        }

        private static int ResolveTemplateTokenId(
            string tokenName,
            JsonElement? ppSpecialTokens,
            IReadOnlyDictionary<string, int> specialTokens,
            IReadOnlyList<(string Piece, float Score)> vocab)
        {
            if (ppSpecialTokens is JsonElement st &&
                st.TryGetProperty(tokenName, out JsonElement entry) &&
                entry.TryGetProperty("ids", out JsonElement ids) &&
                ids.ValueKind == JsonValueKind.Array &&
                ids.GetArrayLength() > 0)
            {
                if (ids[0].ValueKind != JsonValueKind.Number)
                {
                    throw new InvalidDataException($"The tokenizer.json post_processor special token '{tokenName}' has a non-numeric id.");
                }

                int id = ids[0].GetInt32();

                // Validate the id maps back to the referenced token (via added tokens or the vocab), mirroring
                // AddProcessorAffix, so an inconsistent file cannot emit an id whose decoded token differs.
                bool consistent = (specialTokens.TryGetValue(tokenName, out int mappedId) && mappedId == id)
                    || (id >= 0 && id < vocab.Count && vocab[id].Piece == tokenName);
                if (!consistent)
                {
                    throw new InvalidDataException($"The tokenizer.json post_processor special token '{tokenName}' maps to id {id}, which does not match the vocabulary or added tokens.");
                }

                return id;
            }

            if (specialTokens.TryGetValue(tokenName, out int specialId))
            {

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Fix special_tokens so each entry's ids is an array of numbers, e.g. "</s>": {"ids": [2], "type_id": 0}
2. Regenerate tokenizer.json with HuggingFace tokenizers save_pretrained
3. Re-download the original tokenizer.json from the model hub
4. Pre-validate special_tokens entries: id property is a string and ids[0].ValueKind == Number

Example fix

// before
"</s>": {"id": "</s>", "ids": ["2"]}
// after
"</s>": {"id": "</s>", "ids": [2]}

var ids = doc.RootElement.GetProperty("post_processor")
    .GetProperty("special_tokens")[token].GetProperty("ids");
if (ids[0].ValueKind != JsonValueKind.Number) throw new FormatException("ids[0] must be numeric");

static bool HasNumericId(JsonElement specialToken) =>
    specialToken.TryGetProperty("ids", out var ids) && ids.ValueKind == JsonValueKind.Array &&
    ids.GetArrayLength() > 0 && ids[0].ValueKind == JsonValueKind.Number;

try { tok = SentencePieceTokenizer.Create(...); }
catch (InvalidDataException ex) when (ex.Message.Contains("non-numeric id"))
{ /* fix special_tokens or re-export */ }

When it happens

Trigger: tokenizer.json where post_processor.special_tokens[tokenName].ids[0] is a string, null, or other non-number (e.g. "ids": ["2"]); files edited so the ids array no longer matches the HuggingFace format [numericId].

Common situations: Serializers that stringify numbers; hand-edited special_tokens sections; cross-converted tokenizer configs from other tooling.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

• A post_processor template 'SpecialToken.id' must be a string
• Replace normalizer 'String' pattern must be a string.
• unknown option type
• The content of the vocabulary file '{vocabFile}' is not vali
• Problems met when parsing JSON vocabulary object.{Environmen

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/696c589365921242. Report an issue: GitHub.