dotnet/machinelearning · error · InvalidDataException

The tokenizer.json does not contain a 'model' property.

Error message

The tokenizer.json does not contain a 'model' property.

What it means

CreateFromTokenizerJson parses the JSON and requires a top-level 'model' property, throwing InvalidDataException when it is absent. The model property holds the SentencePiece vocabulary/parameters, so the file cannot be interpreted without it.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs:579

        /// </remarks>
        public static SentencePieceTokenizer CreateFromTokenizerJson(
            Stream tokenizerJsonStream,
            bool addBeginningOfSentence = true,
            bool addEndOfSentence = false,
            IReadOnlyDictionary<string, int>? specialTokens = null)
        {
            if (tokenizerJsonStream is null)
            {
                throw new ArgumentNullException(nameof(tokenizerJsonStream));
            }

            using JsonDocument doc = JsonDocument.Parse(tokenizerJsonStream);
            JsonElement root = doc.RootElement;

            // Validate model type
            if (!root.TryGetProperty("model", out JsonElement modelElement))
            {
                throw new InvalidDataException("The tokenizer.json does not contain a 'model' property.");
            }

            if (modelElement.ValueKind != JsonValueKind.Object)
            {
                throw new InvalidDataException("The tokenizer.json 'model' property must be a JSON object.");
            }

            // Validate the model is Unigram. Older tokenizer.json files (e.g. xlm-roberta-base, albert) omit the
            // model "type" entirely; treat a model that has a "vocab" but no BPE "merges" as Unigram, which matches
            // how the Hugging Face loaders disambiguate these files.
            if (modelElement.TryGetProperty("type", out JsonElement modelTypeElement) &&
                modelTypeElement.ValueKind == JsonValueKind.String)
            {
                if (!string.Equals(modelTypeElement.GetString(), "Unigram", StringComparison.OrdinalIgnoreCase))
                {
                    throw new InvalidDataException($"Expected model type 'Unigram' but found '{modelTypeElement.GetString()}'.");
                }
            }

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Point the API at a genuine, complete Hugging Face tokenizer.json containing a top-level 'model' object.
2. Validate the JSON shape before calling: parse it and check root.TryGetProperty("model", ...).
3. Re-download tokenizer.json from the correct model repository.
4. Confirm you're not passing tokenizer_config.json or special_tokens_map.json.

Example fix

// before
var tok = SentencePieceTokenizer.CreateFromTokenizerJson(File.OpenRead("tokenizer_config.json"));
// after
var tok = SentencePieceTokenizer.CreateFromTokenizerJson(File.OpenRead("tokenizer.json")); // correct file with 'model'

using var probe = JsonDocument.Parse(File.ReadAllBytes(path));
if (!probe.RootElement.TryGetProperty("model", out _))
    throw new InvalidDataException($"{path} is not a Hugging Face tokenizer.json with a 'model' property.");

static bool HasModelProperty(JsonElement root) => root.ValueKind == JsonValueKind.Object && root.TryGetProperty("model", out var m) && m.ValueKind == JsonValueKind.Object;

try { var tok = SentencePieceTokenizer.CreateFromTokenizerJson(stream); } catch (InvalidDataException ex) { throw new InvalidOperationException("Invalid or wrong tokenizer.json file.", ex); }

When it happens

Trigger: Passing a tokenizer.json (or any JSON file) that lacks a root-level 'model' object — e.g. a truncated download, a different config file, or a hand-written JSON with the model section under another key.

Common situations: Downloading tokenizer.json from the wrong HF repo file (e.g. tokenizer_config.json instead), copying a partially written file, or a stream that isn't tokenizer.json at all.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• The tokenizer.json 'model' property must be a JSON object.
• An 'added_tokens' entry must have a string 'content' and a n

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/a991fc56dfaab942. Report an issue: GitHub.