dotnet/machinelearning · error · InvalidDataException

Each entry in 'model.vocab' must be a [string piece, number

Error message

Each entry in 'model.vocab' must be a [string piece, number score] pair.

What it means

Vocab entries pass the array-shape check, but element 0 must be a JSON string (the piece) and element 1 a JSON number (the score). Entries with a non-string piece or non-number score are rejected with InvalidDataException.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs:637

                                byteFallbackElement.ValueKind == JsonValueKind.True;

            if (!modelElement.TryGetProperty("vocab", out JsonElement vocabElement) ||
                vocabElement.ValueKind != JsonValueKind.Array)
            {
                throw new InvalidDataException("The tokenizer.json model does not contain a valid 'vocab' array.");
            }

            List<(string Piece, float Score)> vocab = new List<(string Piece, float Score)>(vocabElement.GetArrayLength());
            foreach (JsonElement entry in vocabElement.EnumerateArray())
            {
                if (entry.ValueKind != JsonValueKind.Array || entry.GetArrayLength() < 2)
                {
                    throw new InvalidDataException("Each entry in 'model.vocab' must be a [piece, score] array.");
                }

                if (entry[0].ValueKind != JsonValueKind.String || entry[1].ValueKind != JsonValueKind.Number)
                {
                    throw new InvalidDataException("Each entry in 'model.vocab' must be a [string piece, number score] pair.");
                }

                string? piece = entry[0].GetString();
                if (piece is null)
                {
                    throw new InvalidDataException("A piece string in 'model.vocab' is null.");
                }

                vocab.Add((piece, entry[1].GetSingle()));
            }

            if (unkIsNull)
            {
                // Without an unknown token the only way to represent out-of-vocabulary input is byte fallback; a model
                // with neither cannot encode OOV text, so reject that combination up front rather than emitting an
                // invalid token id at encode time.
                if (!byteFallback)
                {

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Ensure each entry is exactly ["<piece string>", <float score>].
2. Fix the exporter so scores are written as JSON numbers, not quoted strings.
3. Validate the tokenizer.json against the tokenizers library schema before loading.

Example fix

// before
"vocab": [["a", "-1.0"], [0, -2.0]]
// after
"vocab": [["a", -1.0], ["b", -2.0]]

foreach (var e in model.GetProperty("vocab").EnumerateArray())
    if (e[0].ValueKind != JsonValueKind.String || e[1].ValueKind != JsonValueKind.Number)
        throw new InvalidOperationException("Vocab entries must be [string, number].");

try { var tok = SentencePieceTokenizer.CreateFromTokenizerJson(stream); }
catch (InvalidDataException ex) when (ex.Message.Contains("[string piece, number score]")) { /* reject malformed exporter output */ }

When it happens

Trigger: CreateFromTokenizerJson with vocab entries like [0, 0.0] (numeric piece), ["abc", "-1.0"] (string score), or [null, -1.0].

Common situations: Scores serialized as strings by a custom exporter; pieces replaced by ids after a lossy conversion; hand-assembled vocab arrays mixing formats.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• The tokenizer.json model 'unk_id' property must be a number
• Expected model type 'Unigram' but found '{modelTypeElement.G
• The tokenizer.json 'model' has no 'type' and contains 'merge
• The tokenizer.json model does not contain an 'unk_id' proper
• The tokenizer.json model does not contain a valid 'vocab' ar

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/f79059efdbeef591. Report an issue: GitHub.