dotnet/machinelearning · error · InvalidDataException

Each entry in 'model.vocab' must be a [piece, score] array.

Error message

Each entry in 'model.vocab' must be a [piece, score] array.

What it means

Each model.vocab entry must itself be an array of at least two elements: the piece string and its float score. An entry that is not an array, or has fewer than 2 elements, is rejected with InvalidDataException.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs:632

            }

            int unkId = unkIsNull ? -1 : unkIdElement.GetInt32();

            bool byteFallback = modelElement.TryGetProperty("byte_fallback", out JsonElement byteFallbackElement) &&
                                byteFallbackElement.ValueKind == JsonValueKind.True;

            if (!modelElement.TryGetProperty("vocab", out JsonElement vocabElement) ||
                vocabElement.ValueKind != JsonValueKind.Array)
            {
                throw new InvalidDataException("The tokenizer.json model does not contain a valid 'vocab' array.");
            }

            List<(string Piece, float Score)> vocab = new List<(string Piece, float Score)>(vocabElement.GetArrayLength());
            foreach (JsonElement entry in vocabElement.EnumerateArray())
            {
                if (entry.ValueKind != JsonValueKind.Array || entry.GetArrayLength() < 2)
                {
                    throw new InvalidDataException("Each entry in 'model.vocab' must be a [piece, score] array.");
                }

                if (entry[0].ValueKind != JsonValueKind.String || entry[1].ValueKind != JsonValueKind.Number)
                {
                    throw new InvalidDataException("Each entry in 'model.vocab' must be a [string piece, number score] pair.");
                }

                string? piece = entry[0].GetString();
                if (piece is null)
                {
                    throw new InvalidDataException("A piece string in 'model.vocab' is null.");
                }

                vocab.Add((piece, entry[1].GetSingle()));
            }

            if (unkIsNull)
            {

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Rewrite each vocab entry as a two-element ["piece", score] array.
2. When converting a piece->id map, invert it and attach a score (Unigram log-probabilities from the original .model file).
3. Re-export tokenizer.json from the original SentencePiece model instead of hand-converting.

Example fix

// before
"vocab": [["<unk>"], "a"]
// after
"vocab": [["<unk>", 0.0], ["a", -1.0]]

foreach (var e in model.GetProperty("vocab").EnumerateArray())
    if (e.ValueKind != JsonValueKind.Array || e.GetArrayLength() < 2)
        throw new InvalidOperationException("Each vocab entry must be [piece, score].");

try { return SentencePieceTokenizer.CreateFromTokenizerJson(stream); }
catch (InvalidDataException ex) when (ex.Message.Contains("[piece, score]")) { /* fail fast, fix vocab serialization */ }

When it happens

Trigger: CreateFromTokenizerJson with a vocab entry such as "abc" (bare string), ["abc"] (single element), or {} inside the vocab array.

Common situations: Converting from a WordPiece-style vocab (string->id map) without transforming entries into [piece, score] pairs; manually built or partially edited vocab arrays; corrupted exports.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Expected model type 'Unigram' but found '{modelTypeElement.G
• The tokenizer.json 'model' has no 'type' and contains 'merge
• The tokenizer.json model does not contain an 'unk_id' proper
• The tokenizer.json model 'unk_id' property must be a number
• The tokenizer.json model does not contain a valid 'vocab' ar

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/345138f16c6e9476. Report an issue: GitHub.