{"record":{"id":"fc84ad8911da1ebf","repo":"dotnet/machinelearning","slug":"the-tokenizer-json-model-does-not-contain-a-valid","errorCode":null,"errorMessage":"The tokenizer.json model does not contain a valid 'vocab' array.","messagePattern":"The tokenizer\\.json model does not contain a valid 'vocab' array\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs","lineNumber":624,"sourceCode":"            }\n\n            // HF permits a null unk_id, meaning the model has no unknown token; represent that as -1 and validate the\n            // byte_fallback pairing below (a number is validated against the vocabulary once it has been parsed).\n            bool unkIsNull = unkIdElement.ValueKind == JsonValueKind.Null;\n            if (!unkIsNull && unkIdElement.ValueKind != JsonValueKind.Number)\n            {\n                throw new InvalidDataException(\"The tokenizer.json model 'unk_id' property must be a number or null.\");\n            }\n\n            int unkId = unkIsNull ? -1 : unkIdElement.GetInt32();\n\n            bool byteFallback = modelElement.TryGetProperty(\"byte_fallback\", out JsonElement byteFallbackElement) &&\n                                byteFallbackElement.ValueKind == JsonValueKind.True;\n\n            if (!modelElement.TryGetProperty(\"vocab\", out JsonElement vocabElement) ||\n                vocabElement.ValueKind != JsonValueKind.Array)\n            {\n                throw new InvalidDataException(\"The tokenizer.json model does not contain a valid 'vocab' array.\");\n            }\n\n            List<(string Piece, float Score)> vocab = new List<(string Piece, float Score)>(vocabElement.GetArrayLength());\n            foreach (JsonElement entry in vocabElement.EnumerateArray())\n            {\n                if (entry.ValueKind != JsonValueKind.Array || entry.GetArrayLength() < 2)\n                {\n                    throw new InvalidDataException(\"Each entry in 'model.vocab' must be a [piece, score] array.\");\n                }\n\n                if (entry[0].ValueKind != JsonValueKind.String || entry[1].ValueKind != JsonValueKind.Number)\n                {\n                    throw new InvalidDataException(\"Each entry in 'model.vocab' must be a [string piece, number score] pair.\");\n                }\n\n                string? piece = entry[0].GetString();\n                if (piece is null)\n                {","sourceCodeStart":606,"sourceCodeEnd":642,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs#L606-L642","documentation":"A Unigram model's vocabulary is a JSON array of [piece, score] pairs under model.vocab. When that property is absent or not an array, the loader cannot build the vocabulary and throws InvalidDataException.","triggerScenarios":"CreateFromTokenizerJson where the model object has no 'vocab' key, or 'vocab' is an object/string/null instead of an array of pairs.","commonSituations":"tokenizer.json for model formats with different vocab layouts (e.g. WordPiece's object-shaped vocab pasted into a Unigram file); truncated or corrupted tokenizer.json downloads; hand-built configs.","solutions":["Provide model.vocab as an array of [string, number] pairs ordered by token id.","Re-export the tokenizer.json from the original model using the tokenizers library.","Verify the file is complete and not truncated (check array brackets at end of file)."],"exampleFix":"// before\n\"model\": { \"type\": \"Unigram\", \"unk_id\": 0, \"vocab\": { \"<unk>\": 0 } }\n// after\n\"model\": { \"type\": \"Unigram\", \"unk_id\": 0, \"vocab\": [[\"<unk>\", 0.0], [\"a\", -1.0]] }","handlingStrategy":"validation","validationCode":"var model = JsonDocument.Parse(tokenizerJson).RootElement.GetProperty(\"model\");\nif (!model.TryGetProperty(\"vocab\", out var v) || v.ValueKind != JsonValueKind.Array || v.GetArrayLength() == 0)\n    throw new InvalidOperationException(\"model.vocab must be a non-empty array of [piece, score] pairs.\");","typeGuard":null,"tryCatchPattern":"try { var tok = SentencePieceTokenizer.CreateFromTokenizerJson(stream); }\ncatch (InvalidDataException ex) when (ex.Message.Contains(\"'vocab'\")) { log.LogError(ex, \"Invalid or missing vocab\"); throw; }","preventionTips":["Verify tokenizer.json integrity (size/hash) after download or build.","Only feed Unigram-format vocab (array of pairs) to this loader."],"tags":["tokenizer","json-validation","missing-vocab"],"backgroundTag":"missing-required-config-field","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}