dotnet/machinelearning · error · InvalidDataException

Expected model type 'Unigram' but found '{modelTypeElement.G

Error message

Expected model type 'Unigram' but found '{modelTypeElement.GetString()}'.

What it means

CreateFromTokenizerJson only supports Hugging Face Unigram (SentencePiece-style) models. When tokenizer.json's 'model' object has a 'type' field whose value is not 'Unigram' (case-insensitive), this InvalidDataException is thrown to reject BPE/WordPiece/WordLevel and other model types that this factory cannot load.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs:595

            if (!root.TryGetProperty("model", out JsonElement modelElement))
            {
                throw new InvalidDataException("The tokenizer.json does not contain a 'model' property.");
            }

            if (modelElement.ValueKind != JsonValueKind.Object)
            {
                throw new InvalidDataException("The tokenizer.json 'model' property must be a JSON object.");
            }

            // Validate the model is Unigram. Older tokenizer.json files (e.g. xlm-roberta-base, albert) omit the
            // model "type" entirely; treat a model that has a "vocab" but no BPE "merges" as Unigram, which matches
            // how the Hugging Face loaders disambiguate these files.
            if (modelElement.TryGetProperty("type", out JsonElement modelTypeElement) &&
                modelTypeElement.ValueKind == JsonValueKind.String)
            {
                if (!string.Equals(modelTypeElement.GetString(), "Unigram", StringComparison.OrdinalIgnoreCase))
                {
                    throw new InvalidDataException($"Expected model type 'Unigram' but found '{modelTypeElement.GetString()}'.");
                }
            }
            else if (modelElement.TryGetProperty("merges", out _))
            {
                throw new InvalidDataException("The tokenizer.json 'model' has no 'type' and contains 'merges'; this factory only supports 'Unigram' models.");
            }

            if (!modelElement.TryGetProperty("unk_id", out JsonElement unkIdElement))
            {
                throw new InvalidDataException("The tokenizer.json model does not contain an 'unk_id' property.");
            }

            // HF permits a null unk_id, meaning the model has no unknown token; represent that as -1 and validate the
            // byte_fallback pairing below (a number is validated against the vocabulary once it has been parsed).
            bool unkIsNull = unkIdElement.ValueKind == JsonValueKind.Null;
            if (!unkIsNull && unkIdElement.ValueKind != JsonValueKind.Number)
            {
                throw new InvalidDataException("The tokenizer.json model 'unk_id' property must be a number or null.");

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Use the correct tokenizer factory for the model type (e.g. the BPE/WordPiece tokenizer classes in Microsoft.ML.Tokenizers) instead of SentencePieceTokenizer.
2. Obtain a tokenizer.json produced from a genuine SentencePiece Unigram model (e.g. Llama, T5) rather than a BPE/WordPiece one.
3. Fix the model.type field if it is a typo; it must be exactly 'Unigram' (case-insensitive).

Example fix

// before
"model": { "type": "BPE", "vocab": ..., "merges": ... }
// after
"model": { "type": "Unigram", "unk_id": 0, "vocab": [["<unk>", 0.0], ...] }

using var doc = JsonDocument.Parse(tokenizerJson);
var model = doc.RootElement.GetProperty("model");
if (model.TryGetProperty("type", out var t) &&
    !string.Equals(t.GetString(), "Unigram", StringComparison.OrdinalIgnoreCase))
    throw new InvalidOperationException($"Model type '{t.GetString()}' not supported by SentencePieceTokenizer.");

try { var tok = SentencePieceTokenizer.CreateFromTokenizerJson(stream); }
catch (InvalidDataException ex) { log.LogError(ex, "Unsupported tokenizer model type"); throw new UnsupportedTokenizerException(...); }

When it happens

Trigger: Calling SentencePieceTokenizer.CreateFromTokenizerJson (or the HF tokenizer.json loader path) with a tokenizer.json whose model.type is 'BPE', 'WordPiece', 'WordLevel', 'FastBPE', or any misspelled/alternate casing variant other than 'Unigram'.

Common situations: Pointing the factory at a tokenizer.json exported from a GPT-2/Llama-BPE or BERT WordPiece tokenizer instead of a SentencePiece/Unigram model (e.g. Llama 2, T5, XLNet are Unigram; GPT-2/Phi are BPE); typos in the type field; hand-edited tokenizer.json files.

Understand the failure class

Background: "is not a compatible type" / "cannot merge" errors: when a value's type doesn't match what the library requires — this error's family across 65 libraries.

Related errors

• The tokenizer.json 'model' has no 'type' and contains 'merge
• The tokenizer.json model does not contain an 'unk_id' proper
• The tokenizer.json model 'unk_id' property must be a number
• The tokenizer.json model does not contain a valid 'vocab' ar
• Each entry in 'model.vocab' must be a [piece, score] array.

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/23fbcca82bb588da. Report an issue: GitHub.