{"record":{"id":"345138f16c6e9476","repo":"dotnet/machinelearning","slug":"each-entry-in-model-vocab-must-be-a-piece-scor","errorCode":null,"errorMessage":"Each entry in 'model.vocab' must be a [piece, score] array.","messagePattern":"Each entry in 'model\\.vocab' must be a \\[piece, score\\] array\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs","lineNumber":632,"sourceCode":"            }\n\n            int unkId = unkIsNull ? -1 : unkIdElement.GetInt32();\n\n            bool byteFallback = modelElement.TryGetProperty(\"byte_fallback\", out JsonElement byteFallbackElement) &&\n                                byteFallbackElement.ValueKind == JsonValueKind.True;\n\n            if (!modelElement.TryGetProperty(\"vocab\", out JsonElement vocabElement) ||\n                vocabElement.ValueKind != JsonValueKind.Array)\n            {\n                throw new InvalidDataException(\"The tokenizer.json model does not contain a valid 'vocab' array.\");\n            }\n\n            List<(string Piece, float Score)> vocab = new List<(string Piece, float Score)>(vocabElement.GetArrayLength());\n            foreach (JsonElement entry in vocabElement.EnumerateArray())\n            {\n                if (entry.ValueKind != JsonValueKind.Array || entry.GetArrayLength() < 2)\n                {\n                    throw new InvalidDataException(\"Each entry in 'model.vocab' must be a [piece, score] array.\");\n                }\n\n                if (entry[0].ValueKind != JsonValueKind.String || entry[1].ValueKind != JsonValueKind.Number)\n                {\n                    throw new InvalidDataException(\"Each entry in 'model.vocab' must be a [string piece, number score] pair.\");\n                }\n\n                string? piece = entry[0].GetString();\n                if (piece is null)\n                {\n                    throw new InvalidDataException(\"A piece string in 'model.vocab' is null.\");\n                }\n\n                vocab.Add((piece, entry[1].GetSingle()));\n            }\n\n            if (unkIsNull)\n            {","sourceCodeStart":614,"sourceCodeEnd":650,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs#L614-L650","documentation":"Each model.vocab entry must itself be an array of at least two elements: the piece string and its float score. An entry that is not an array, or has fewer than 2 elements, is rejected with InvalidDataException.","triggerScenarios":"CreateFromTokenizerJson with a vocab entry such as \"abc\" (bare string), [\"abc\"] (single element), or {} inside the vocab array.","commonSituations":"Converting from a WordPiece-style vocab (string->id map) without transforming entries into [piece, score] pairs; manually built or partially edited vocab arrays; corrupted exports.","solutions":["Rewrite each vocab entry as a two-element [\"piece\", score] array.","When converting a piece->id map, invert it and attach a score (Unigram log-probabilities from the original .model file).","Re-export tokenizer.json from the original SentencePiece model instead of hand-converting."],"exampleFix":"// before\n\"vocab\": [[\"<unk>\"], \"a\"]\n// after\n\"vocab\": [[\"<unk>\", 0.0], [\"a\", -1.0]]","handlingStrategy":"validation","validationCode":"foreach (var e in model.GetProperty(\"vocab\").EnumerateArray())\n    if (e.ValueKind != JsonValueKind.Array || e.GetArrayLength() < 2)\n        throw new InvalidOperationException(\"Each vocab entry must be [piece, score].\");","typeGuard":null,"tryCatchPattern":"try { return SentencePieceTokenizer.CreateFromTokenizerJson(stream); }\ncatch (InvalidDataException ex) when (ex.Message.Contains(\"[piece, score]\")) { /* fail fast, fix vocab serialization */ }","preventionTips":["When converting from piece->id maps, transform entries into [piece, score] pairs.","Round-trip tokenizer.json through the tokenizers library to normalize shapes."],"tags":["tokenizer","json-validation","malformed-vocab-entry"],"backgroundTag":"schema-validation-failed","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}