dotnet/machinelearning · error · InvalidDataException

The tokenizer.json 'model' has no 'type' and contains 'merge

Error message

The tokenizer.json 'model' has no 'type' and contains 'merges'; this factory only supports 'Unigram' models.

What it means

If the 'model' object has no 'type' field but does contain a 'merges' array, the loader infers it is a BPE-style model, which this factory does not support. The throw makes the inference explicit instead of failing later with a confusing vocabulary error.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs:600

            if (modelElement.ValueKind != JsonValueKind.Object)
            {
                throw new InvalidDataException("The tokenizer.json 'model' property must be a JSON object.");
            }

            // Validate the model is Unigram. Older tokenizer.json files (e.g. xlm-roberta-base, albert) omit the
            // model "type" entirely; treat a model that has a "vocab" but no BPE "merges" as Unigram, which matches
            // how the Hugging Face loaders disambiguate these files.
            if (modelElement.TryGetProperty("type", out JsonElement modelTypeElement) &&
                modelTypeElement.ValueKind == JsonValueKind.String)
            {
                if (!string.Equals(modelTypeElement.GetString(), "Unigram", StringComparison.OrdinalIgnoreCase))
                {
                    throw new InvalidDataException($"Expected model type 'Unigram' but found '{modelTypeElement.GetString()}'.");
                }
            }
            else if (modelElement.TryGetProperty("merges", out _))
            {
                throw new InvalidDataException("The tokenizer.json 'model' has no 'type' and contains 'merges'; this factory only supports 'Unigram' models.");
            }

            if (!modelElement.TryGetProperty("unk_id", out JsonElement unkIdElement))
            {
                throw new InvalidDataException("The tokenizer.json model does not contain an 'unk_id' property.");
            }

            // HF permits a null unk_id, meaning the model has no unknown token; represent that as -1 and validate the
            // byte_fallback pairing below (a number is validated against the vocabulary once it has been parsed).
            bool unkIsNull = unkIdElement.ValueKind == JsonValueKind.Null;
            if (!unkIsNull && unkIdElement.ValueKind != JsonValueKind.Number)
            {
                throw new InvalidDataException("The tokenizer.json model 'unk_id' property must be a number or null.");
            }

            int unkId = unkIsNull ? -1 : unkIdElement.GetInt32();

            bool byteFallback = modelElement.TryGetProperty("byte_fallback", out JsonElement byteFallbackElement) &&

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Add "type": "Unigram" only if the model truly is Unigram; otherwise use the BPE tokenizer loader instead.
2. Regenerate tokenizer.json with a complete model section (convert_tokenizer from tokenizers lib preserves the type field).
3. Switch to a tokenizer.json for a SentencePiece/Unigram model.

Example fix

// before
"model": { "vocab": [...], "merges": [...] }
// after (if truly Unigram)
"model": { "type": "Unigram", "unk_id": 0, "vocab": [...] }

var model = JsonDocument.Parse(tokenizerJson).RootElement.GetProperty("model");
if (!model.TryGetProperty("type", out _) && model.TryGetProperty("merges", out _))
    throw new InvalidOperationException("tokenizer.json is BPE-style; use a BPE loader.");

try { return SentencePieceTokenizer.CreateFromTokenizerJson(stream); }
catch (InvalidDataException ex) when (ex.Message.Contains("'merges'")) { /* fall back to BPE tokenizer loader */ }

When it happens

Trigger: CreateFromTokenizerJson given a tokenizer.json whose model object omits 'type' but includes 'merges' (the BPE signature).

Common situations: Older or hand-trimmed tokenizer.json exports that dropped the type field; GPT-2-style BPE tokenizers; minimal tokenizer.json snippets copied from blog posts.

Understand the failure class

Background: "is not a compatible type" / "cannot merge" errors: when a value's type doesn't match what the library requires — this error's family across 65 libraries.

Related errors

• Expected model type 'Unigram' but found '{modelTypeElement.G
• The tokenizer.json model does not contain an 'unk_id' proper
• The tokenizer.json model 'unk_id' property must be a number
• The tokenizer.json model does not contain a valid 'vocab' ar
• Each entry in 'model.vocab' must be a [piece, score] array.

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/482d886220ed417a. Report an issue: GitHub.