{"record":{"id":"f79059efdbeef591","repo":"dotnet/machinelearning","slug":"each-entry-in-model-vocab-must-be-a-string-piec","errorCode":null,"errorMessage":"Each entry in 'model.vocab' must be a [string piece, number score] pair.","messagePattern":"Each entry in 'model\\.vocab' must be a \\[string piece, number score\\] pair\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs","lineNumber":637,"sourceCode":"                                byteFallbackElement.ValueKind == JsonValueKind.True;\n\n            if (!modelElement.TryGetProperty(\"vocab\", out JsonElement vocabElement) ||\n                vocabElement.ValueKind != JsonValueKind.Array)\n            {\n                throw new InvalidDataException(\"The tokenizer.json model does not contain a valid 'vocab' array.\");\n            }\n\n            List<(string Piece, float Score)> vocab = new List<(string Piece, float Score)>(vocabElement.GetArrayLength());\n            foreach (JsonElement entry in vocabElement.EnumerateArray())\n            {\n                if (entry.ValueKind != JsonValueKind.Array || entry.GetArrayLength() < 2)\n                {\n                    throw new InvalidDataException(\"Each entry in 'model.vocab' must be a [piece, score] array.\");\n                }\n\n                if (entry[0].ValueKind != JsonValueKind.String || entry[1].ValueKind != JsonValueKind.Number)\n                {\n                    throw new InvalidDataException(\"Each entry in 'model.vocab' must be a [string piece, number score] pair.\");\n                }\n\n                string? piece = entry[0].GetString();\n                if (piece is null)\n                {\n                    throw new InvalidDataException(\"A piece string in 'model.vocab' is null.\");\n                }\n\n                vocab.Add((piece, entry[1].GetSingle()));\n            }\n\n            if (unkIsNull)\n            {\n                // Without an unknown token the only way to represent out-of-vocabulary input is byte fallback; a model\n                // with neither cannot encode OOV text, so reject that combination up front rather than emitting an\n                // invalid token id at encode time.\n                if (!byteFallback)\n                {","sourceCodeStart":619,"sourceCodeEnd":655,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs#L619-L655","documentation":"Vocab entries pass the array-shape check, but element 0 must be a JSON string (the piece) and element 1 a JSON number (the score). Entries with a non-string piece or non-number score are rejected with InvalidDataException.","triggerScenarios":"CreateFromTokenizerJson with vocab entries like [0, 0.0] (numeric piece), [\"abc\", \"-1.0\"] (string score), or [null, -1.0].","commonSituations":"Scores serialized as strings by a custom exporter; pieces replaced by ids after a lossy conversion; hand-assembled vocab arrays mixing formats.","solutions":["Ensure each entry is exactly [\"<piece string>\", <float score>].","Fix the exporter so scores are written as JSON numbers, not quoted strings.","Validate the tokenizer.json against the tokenizers library schema before loading."],"exampleFix":"// before\n\"vocab\": [[\"a\", \"-1.0\"], [0, -2.0]]\n// after\n\"vocab\": [[\"a\", -1.0], [\"b\", -2.0]]","handlingStrategy":"validation","validationCode":"foreach (var e in model.GetProperty(\"vocab\").EnumerateArray())\n    if (e[0].ValueKind != JsonValueKind.String || e[1].ValueKind != JsonValueKind.Number)\n        throw new InvalidOperationException(\"Vocab entries must be [string, number].\");","typeGuard":null,"tryCatchPattern":"try { var tok = SentencePieceTokenizer.CreateFromTokenizerJson(stream); }\ncatch (InvalidDataException ex) when (ex.Message.Contains(\"[string piece, number score]\")) { /* reject malformed exporter output */ }","preventionTips":["Ensure scores are emitted as JSON numbers, not strings.","Add a schema-validation step for tokenizer.json in your build pipeline."],"tags":["tokenizer","json-validation","type-mismatch"],"backgroundTag":"schema-validation-failed","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}