dotnet/machinelearning · error · InvalidDataException

The tokenizer.json model does not contain a valid 'vocab' ar

Error message

The tokenizer.json model does not contain a valid 'vocab' array.

What it means

A Unigram model's vocabulary is a JSON array of [piece, score] pairs under model.vocab. When that property is absent or not an array, the loader cannot build the vocabulary and throws InvalidDataException.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs:624

            }

            // HF permits a null unk_id, meaning the model has no unknown token; represent that as -1 and validate the
            // byte_fallback pairing below (a number is validated against the vocabulary once it has been parsed).
            bool unkIsNull = unkIdElement.ValueKind == JsonValueKind.Null;
            if (!unkIsNull && unkIdElement.ValueKind != JsonValueKind.Number)
            {
                throw new InvalidDataException("The tokenizer.json model 'unk_id' property must be a number or null.");
            }

            int unkId = unkIsNull ? -1 : unkIdElement.GetInt32();

            bool byteFallback = modelElement.TryGetProperty("byte_fallback", out JsonElement byteFallbackElement) &&
                                byteFallbackElement.ValueKind == JsonValueKind.True;

            if (!modelElement.TryGetProperty("vocab", out JsonElement vocabElement) ||
                vocabElement.ValueKind != JsonValueKind.Array)
            {
                throw new InvalidDataException("The tokenizer.json model does not contain a valid 'vocab' array.");
            }

            List<(string Piece, float Score)> vocab = new List<(string Piece, float Score)>(vocabElement.GetArrayLength());
            foreach (JsonElement entry in vocabElement.EnumerateArray())
            {
                if (entry.ValueKind != JsonValueKind.Array || entry.GetArrayLength() < 2)
                {
                    throw new InvalidDataException("Each entry in 'model.vocab' must be a [piece, score] array.");
                }

                if (entry[0].ValueKind != JsonValueKind.String || entry[1].ValueKind != JsonValueKind.Number)
                {
                    throw new InvalidDataException("Each entry in 'model.vocab' must be a [string piece, number score] pair.");
                }

                string? piece = entry[0].GetString();
                if (piece is null)
                {

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Provide model.vocab as an array of [string, number] pairs ordered by token id.
2. Re-export the tokenizer.json from the original model using the tokenizers library.
3. Verify the file is complete and not truncated (check array brackets at end of file).

Example fix

// before
"model": { "type": "Unigram", "unk_id": 0, "vocab": { "<unk>": 0 } }
// after
"model": { "type": "Unigram", "unk_id": 0, "vocab": [["<unk>", 0.0], ["a", -1.0]] }

var model = JsonDocument.Parse(tokenizerJson).RootElement.GetProperty("model");
if (!model.TryGetProperty("vocab", out var v) || v.ValueKind != JsonValueKind.Array || v.GetArrayLength() == 0)
    throw new InvalidOperationException("model.vocab must be a non-empty array of [piece, score] pairs.");

try { var tok = SentencePieceTokenizer.CreateFromTokenizerJson(stream); }
catch (InvalidDataException ex) when (ex.Message.Contains("'vocab'")) { log.LogError(ex, "Invalid or missing vocab"); throw; }

When it happens

Trigger: CreateFromTokenizerJson where the model object has no 'vocab' key, or 'vocab' is an object/string/null instead of an array of pairs.

Common situations: tokenizer.json for model formats with different vocab layouts (e.g. WordPiece's object-shaped vocab pasted into a Unigram file); truncated or corrupted tokenizer.json downloads; hand-built configs.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

• Expected model type 'Unigram' but found '{modelTypeElement.G
• The tokenizer.json 'model' has no 'type' and contains 'merge
• The tokenizer.json model does not contain an 'unk_id' proper
• The tokenizer.json model 'unk_id' property must be a number
• Each entry in 'model.vocab' must be a [piece, score] array.

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/fc84ad8911da1ebf. Report an issue: GitHub.