{"record":{"id":"851cbe1c3e16b13a","repo":"dotnet/machinelearning","slug":"the-tokenizer-json-model-property-must-be-a-json","errorCode":null,"errorMessage":"The tokenizer.json 'model' property must be a JSON object.","messagePattern":"The tokenizer\\.json 'model' property must be a JSON object\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs","lineNumber":584,"sourceCode":"            IReadOnlyDictionary<string, int>? specialTokens = null)\n        {\n            if (tokenizerJsonStream is null)\n            {\n                throw new ArgumentNullException(nameof(tokenizerJsonStream));\n            }\n\n            using JsonDocument doc = JsonDocument.Parse(tokenizerJsonStream);\n            JsonElement root = doc.RootElement;\n\n            // Validate model type\n            if (!root.TryGetProperty(\"model\", out JsonElement modelElement))\n            {\n                throw new InvalidDataException(\"The tokenizer.json does not contain a 'model' property.\");\n            }\n\n            if (modelElement.ValueKind != JsonValueKind.Object)\n            {\n                throw new InvalidDataException(\"The tokenizer.json 'model' property must be a JSON object.\");\n            }\n\n            // Validate the model is Unigram. Older tokenizer.json files (e.g. xlm-roberta-base, albert) omit the\n            // model \"type\" entirely; treat a model that has a \"vocab\" but no BPE \"merges\" as Unigram, which matches\n            // how the Hugging Face loaders disambiguate these files.\n            if (modelElement.TryGetProperty(\"type\", out JsonElement modelTypeElement) &&\n                modelTypeElement.ValueKind == JsonValueKind.String)\n            {\n                if (!string.Equals(modelTypeElement.GetString(), \"Unigram\", StringComparison.OrdinalIgnoreCase))\n                {\n                    throw new InvalidDataException($\"Expected model type 'Unigram' but found '{modelTypeElement.GetString()}'.\");\n                }\n            }\n            else if (modelElement.TryGetProperty(\"merges\", out _))\n            {\n                throw new InvalidDataException(\"The tokenizer.json 'model' has no 'type' and contains 'merges'; this factory only supports 'Unigram' models.\");\n            }\n","sourceCodeStart":566,"sourceCodeEnd":602,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs#L566-L602","documentation":"The 'model' property in tokenizer.json must be a JSON object; CreateFromTokenizerJson throws InvalidDataException when it is some other JSON kind (string, array, number). The library reads vocab/merges/type fields from that object, so a non-object value is unusable.","triggerScenarios":"Loading a tokenizer.json whose model field is malformed — e.g. hand-edited JSON where \"model\" was replaced by a string, a corrupted/truncated write, or a custom export emitting a different structure.","commonSituations":"Manual edits to tokenizer.json, template-based generation emitting a placeholder string, or tools exporting a non-HF-compatible schema.","solutions":["Fix the tokenizer.json so 'model' is a JSON object (e.g. {\"type\": \"Unigram\", \"vocab\": [...]}).","Re-export or re-download the file from the original source.","Validate with JSON Schema or a quick parse check before passing it to the API.","Avoid hand-editing; use the exporting tool's config to change model settings."],"exampleFix":"// before (broken)\n// { \"model\": \"unigram\" }\n// after\n// { \"model\": { \"type\": \"Unigram\", \"unk_id\": 0, \"vocab\": [] } }","handlingStrategy":"validation","validationCode":"using var probe = JsonDocument.Parse(File.ReadAllBytes(path));\nprobe.RootElement.TryGetProperty(\"model\", out var m);\nif (m.ValueKind != JsonValueKind.Object) throw new InvalidDataException(\"'model' must be a JSON object.\");","typeGuard":"static bool IsModelObject(JsonElement m) => m.ValueKind == JsonValueKind.Object;","tryCatchPattern":"try { var tok = SentencePieceTokenizer.CreateFromTokenizerJson(stream); } catch (InvalidDataException ex) { // re-download or regenerate tokenizer.json\n    throw new InvalidOperationException(\"tokenizer.json model section is malformed.\", ex); }","preventionTips":["Never hand-edit tokenizer.json; regenerate via the exporting tool","Run JSON Schema validation on generated tokenizer files","Keep model settings in the trainer/exporter config instead"],"tags":["schema-validation","tokenizer-json","json"],"backgroundTag":"schema-validation-failed","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}