dotnet/machinelearning · error · InvalidDataException

The tokenizer.json model 'unk_id' property must be a number

Error message

The tokenizer.json model 'unk_id' property must be a number or null.

What it means

The 'unk_id' property exists but is neither a JSON number nor null. The loader accepts only an integer vocabulary index or null (meaning 'no unknown token'); any other JSON kind (string, bool, object, array) is rejected with InvalidDataException.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs:613

                    throw new InvalidDataException($"Expected model type 'Unigram' but found '{modelTypeElement.GetString()}'.");
                }
            }
            else if (modelElement.TryGetProperty("merges", out _))
            {
                throw new InvalidDataException("The tokenizer.json 'model' has no 'type' and contains 'merges'; this factory only supports 'Unigram' models.");
            }

            if (!modelElement.TryGetProperty("unk_id", out JsonElement unkIdElement))
            {
                throw new InvalidDataException("The tokenizer.json model does not contain an 'unk_id' property.");
            }

            // HF permits a null unk_id, meaning the model has no unknown token; represent that as -1 and validate the
            // byte_fallback pairing below (a number is validated against the vocabulary once it has been parsed).
            bool unkIsNull = unkIdElement.ValueKind == JsonValueKind.Null;
            if (!unkIsNull && unkIdElement.ValueKind != JsonValueKind.Number)
            {
                throw new InvalidDataException("The tokenizer.json model 'unk_id' property must be a number or null.");
            }

            int unkId = unkIsNull ? -1 : unkIdElement.GetInt32();

            bool byteFallback = modelElement.TryGetProperty("byte_fallback", out JsonElement byteFallbackElement) &&
                                byteFallbackElement.ValueKind == JsonValueKind.True;

            if (!modelElement.TryGetProperty("vocab", out JsonElement vocabElement) ||
                vocabElement.ValueKind != JsonValueKind.Array)
            {
                throw new InvalidDataException("The tokenizer.json model does not contain a valid 'vocab' array.");
            }

            List<(string Piece, float Score)> vocab = new List<(string Piece, float Score)>(vocabElement.GetArrayLength());
            foreach (JsonElement entry in vocabElement.EnumerateArray())
            {
                if (entry.ValueKind != JsonValueKind.Array || entry.GetArrayLength() < 2)
                {

View on GitHub (pinned to 7b76e69cf9)

Solutions

Change unk_id to an unquoted integer, e.g. "unk_id": 0.
Use null if the model has no unknown token, ensuring byte_fallback is true.
Validate the tokenizer.json with the Hugging Face tokenizers library before loading it here.

Example fix

// before
"unk_id": "0"
// after
"unk_id": 0

Defensive patterns

Strategy: validation

Validate before calling

var unkId = model.GetProperty("unk_id");
if (unkId.ValueKind is not (JsonValueKind.Number or JsonValueKind.Null))
    throw new InvalidOperationException("unk_id must be a number or null.");

Try / catch

try { return SentencePieceTokenizer.CreateFromTokenizerJson(stream); }
catch (InvalidDataException ex) when (ex.Message.Contains("must be a number or null")) { /* repair or reject the tokenizer.json */ }

Prevention

Never quote numeric ids when generating tokenizer.json.
Deserialize-then-serialize the JSON with a typed model to normalize kinds.

When it happens

Trigger: CreateFromTokenizerJson with model.unk_id set to e.g. "0" (a string), true, or an object/array.

Common situations: Hand-edited tokenizer.json quoting the id; JSON produced by tools that serialize ids as strings; copy-paste mistakes when constructing the model section manually.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/4b3d6be80f47cfec. Report an issue: GitHub.