dotnet/machinelearning · error · InvalidDataException

The tokenizer.json model does not contain an 'unk_id' proper

Error message

The tokenizer.json model does not contain an 'unk_id' property.

What it means

A Unigram model requires an 'unk_id' property (the vocabulary id of the unknown-token piece, or explicit null). The loader refuses to guess it, so a tokenizer.json whose model lacks this field is rejected with InvalidDataException.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs:605

            // Validate the model is Unigram. Older tokenizer.json files (e.g. xlm-roberta-base, albert) omit the
            // model "type" entirely; treat a model that has a "vocab" but no BPE "merges" as Unigram, which matches
            // how the Hugging Face loaders disambiguate these files.
            if (modelElement.TryGetProperty("type", out JsonElement modelTypeElement) &&
                modelTypeElement.ValueKind == JsonValueKind.String)
            {
                if (!string.Equals(modelTypeElement.GetString(), "Unigram", StringComparison.OrdinalIgnoreCase))
                {
                    throw new InvalidDataException($"Expected model type 'Unigram' but found '{modelTypeElement.GetString()}'.");
                }
            }
            else if (modelElement.TryGetProperty("merges", out _))
            {
                throw new InvalidDataException("The tokenizer.json 'model' has no 'type' and contains 'merges'; this factory only supports 'Unigram' models.");
            }

            if (!modelElement.TryGetProperty("unk_id", out JsonElement unkIdElement))
            {
                throw new InvalidDataException("The tokenizer.json model does not contain an 'unk_id' property.");
            }

            // HF permits a null unk_id, meaning the model has no unknown token; represent that as -1 and validate the
            // byte_fallback pairing below (a number is validated against the vocabulary once it has been parsed).
            bool unkIsNull = unkIdElement.ValueKind == JsonValueKind.Null;
            if (!unkIsNull && unkIdElement.ValueKind != JsonValueKind.Number)
            {
                throw new InvalidDataException("The tokenizer.json model 'unk_id' property must be a number or null.");
            }

            int unkId = unkIsNull ? -1 : unkIdElement.GetInt32();

            bool byteFallback = modelElement.TryGetProperty("byte_fallback", out JsonElement byteFallbackElement) &&
                                byteFallbackElement.ValueKind == JsonValueKind.True;

            if (!modelElement.TryGetProperty("vocab", out JsonElement vocabElement) ||
                vocabElement.ValueKind != JsonValueKind.Array)
            {

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Add "unk_id": <int> to the model object, set to the index of the '<unk>' piece in vocab.
2. Set "unk_id": null if the model genuinely has no unknown token (requires byte_fallback enabled).
3. Re-export the tokenizer with the tokenizers library so all required Unigram fields are present.

Example fix

// before
"model": { "type": "Unigram", "vocab": [["<unk>", 0.0], ...] }
// after
"model": { "type": "Unigram", "unk_id": 0, "vocab": [["<unk>", 0.0], ...] }

var model = JsonDocument.Parse(tokenizerJson).RootElement.GetProperty("model");
if (!model.TryGetProperty("unk_id", out _))
    throw new InvalidOperationException("model.unk_id is required (number or null).");

try { var tok = SentencePieceTokenizer.CreateFromTokenizerJson(stream); }
catch (InvalidDataException ex) when (ex.Message.Contains("unk_id")) { log.LogError(ex, "tokenizer.json missing unk_id"); throw; }

When it happens

Trigger: CreateFromTokenizerJson with a model object that has type 'Unigram' and a 'vocab' but no 'unk_id' key.

Common situations: Hand-written or minimal tokenizer.json files; exports from custom SentencePiece training where the id was stripped; copying only the vocab from a larger file.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

• Expected model type 'Unigram' but found '{modelTypeElement.G
• The tokenizer.json 'model' has no 'type' and contains 'merge
• The tokenizer.json model 'unk_id' property must be a number
• The tokenizer.json model does not contain a valid 'vocab' ar
• Each entry in 'model.vocab' must be a [piece, score] array.

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/2f784c62555322c4. Report an issue: GitHub.