dotnet/machinelearning · error · InvalidDataException

The tokenizer.json 'model' property must be a JSON object.

Error message

The tokenizer.json 'model' property must be a JSON object.

What it means

The 'model' property in tokenizer.json must be a JSON object; CreateFromTokenizerJson throws InvalidDataException when it is some other JSON kind (string, array, number). The library reads vocab/merges/type fields from that object, so a non-object value is unusable.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs:584

            IReadOnlyDictionary<string, int>? specialTokens = null)
        {
            if (tokenizerJsonStream is null)
            {
                throw new ArgumentNullException(nameof(tokenizerJsonStream));
            }

            using JsonDocument doc = JsonDocument.Parse(tokenizerJsonStream);
            JsonElement root = doc.RootElement;

            // Validate model type
            if (!root.TryGetProperty("model", out JsonElement modelElement))
            {
                throw new InvalidDataException("The tokenizer.json does not contain a 'model' property.");
            }

            if (modelElement.ValueKind != JsonValueKind.Object)
            {
                throw new InvalidDataException("The tokenizer.json 'model' property must be a JSON object.");
            }

            // Validate the model is Unigram. Older tokenizer.json files (e.g. xlm-roberta-base, albert) omit the
            // model "type" entirely; treat a model that has a "vocab" but no BPE "merges" as Unigram, which matches
            // how the Hugging Face loaders disambiguate these files.
            if (modelElement.TryGetProperty("type", out JsonElement modelTypeElement) &&
                modelTypeElement.ValueKind == JsonValueKind.String)
            {
                if (!string.Equals(modelTypeElement.GetString(), "Unigram", StringComparison.OrdinalIgnoreCase))
                {
                    throw new InvalidDataException($"Expected model type 'Unigram' but found '{modelTypeElement.GetString()}'.");
                }
            }
            else if (modelElement.TryGetProperty("merges", out _))
            {
                throw new InvalidDataException("The tokenizer.json 'model' has no 'type' and contains 'merges'; this factory only supports 'Unigram' models.");
            }

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Fix the tokenizer.json so 'model' is a JSON object (e.g. {"type": "Unigram", "vocab": [...]}).
2. Re-export or re-download the file from the original source.
3. Validate with JSON Schema or a quick parse check before passing it to the API.
4. Avoid hand-editing; use the exporting tool's config to change model settings.

Example fix

// before (broken)
// { "model": "unigram" }
// after
// { "model": { "type": "Unigram", "unk_id": 0, "vocab": [] } }

using var probe = JsonDocument.Parse(File.ReadAllBytes(path));
probe.RootElement.TryGetProperty("model", out var m);
if (m.ValueKind != JsonValueKind.Object) throw new InvalidDataException("'model' must be a JSON object.");

static bool IsModelObject(JsonElement m) => m.ValueKind == JsonValueKind.Object;

try { var tok = SentencePieceTokenizer.CreateFromTokenizerJson(stream); } catch (InvalidDataException ex) { // re-download or regenerate tokenizer.json
    throw new InvalidOperationException("tokenizer.json model section is malformed.", ex); }

When it happens

Trigger: Loading a tokenizer.json whose model field is malformed — e.g. hand-edited JSON where "model" was replaced by a string, a corrupted/truncated write, or a custom export emitting a different structure.

Common situations: Manual edits to tokenizer.json, template-based generation emitting a placeholder string, or tools exporting a non-HF-compatible schema.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• The tokenizer.json does not contain a 'model' property.
• An 'added_tokens' entry must have a string 'content' and a n
• Failed to deserialize tool calls.
• unknown schema type: {schema.Type}
• unknown type

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/851cbe1c3e16b13a. Report an issue: GitHub.