{"record":{"id":"a991fc56dfaab942","repo":"dotnet/machinelearning","slug":"the-tokenizer-json-does-not-contain-a-model-prop","errorCode":null,"errorMessage":"The tokenizer.json does not contain a 'model' property.","messagePattern":"The tokenizer\\.json does not contain a 'model' property\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs","lineNumber":579,"sourceCode":"        /// </remarks>\n        public static SentencePieceTokenizer CreateFromTokenizerJson(\n            Stream tokenizerJsonStream,\n            bool addBeginningOfSentence = true,\n            bool addEndOfSentence = false,\n            IReadOnlyDictionary<string, int>? specialTokens = null)\n        {\n            if (tokenizerJsonStream is null)\n            {\n                throw new ArgumentNullException(nameof(tokenizerJsonStream));\n            }\n\n            using JsonDocument doc = JsonDocument.Parse(tokenizerJsonStream);\n            JsonElement root = doc.RootElement;\n\n            // Validate model type\n            if (!root.TryGetProperty(\"model\", out JsonElement modelElement))\n            {\n                throw new InvalidDataException(\"The tokenizer.json does not contain a 'model' property.\");\n            }\n\n            if (modelElement.ValueKind != JsonValueKind.Object)\n            {\n                throw new InvalidDataException(\"The tokenizer.json 'model' property must be a JSON object.\");\n            }\n\n            // Validate the model is Unigram. Older tokenizer.json files (e.g. xlm-roberta-base, albert) omit the\n            // model \"type\" entirely; treat a model that has a \"vocab\" but no BPE \"merges\" as Unigram, which matches\n            // how the Hugging Face loaders disambiguate these files.\n            if (modelElement.TryGetProperty(\"type\", out JsonElement modelTypeElement) &&\n                modelTypeElement.ValueKind == JsonValueKind.String)\n            {\n                if (!string.Equals(modelTypeElement.GetString(), \"Unigram\", StringComparison.OrdinalIgnoreCase))\n                {\n                    throw new InvalidDataException($\"Expected model type 'Unigram' but found '{modelTypeElement.GetString()}'.\");\n                }\n            }","sourceCodeStart":561,"sourceCodeEnd":597,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs#L561-L597","documentation":"CreateFromTokenizerJson parses the JSON and requires a top-level 'model' property, throwing InvalidDataException when it is absent. The model property holds the SentencePiece vocabulary/parameters, so the file cannot be interpreted without it.","triggerScenarios":"Passing a tokenizer.json (or any JSON file) that lacks a root-level 'model' object — e.g. a truncated download, a different config file, or a hand-written JSON with the model section under another key.","commonSituations":"Downloading tokenizer.json from the wrong HF repo file (e.g. tokenizer_config.json instead), copying a partially written file, or a stream that isn't tokenizer.json at all.","solutions":["Point the API at a genuine, complete Hugging Face tokenizer.json containing a top-level 'model' object.","Validate the JSON shape before calling: parse it and check root.TryGetProperty(\"model\", ...).","Re-download tokenizer.json from the correct model repository.","Confirm you're not passing tokenizer_config.json or special_tokens_map.json."],"exampleFix":"// before\nvar tok = SentencePieceTokenizer.CreateFromTokenizerJson(File.OpenRead(\"tokenizer_config.json\"));\n// after\nvar tok = SentencePieceTokenizer.CreateFromTokenizerJson(File.OpenRead(\"tokenizer.json\")); // correct file with 'model'","handlingStrategy":"validation","validationCode":"using var probe = JsonDocument.Parse(File.ReadAllBytes(path));\nif (!probe.RootElement.TryGetProperty(\"model\", out _))\n    throw new InvalidDataException($\"{path} is not a Hugging Face tokenizer.json with a 'model' property.\");","typeGuard":"static bool HasModelProperty(JsonElement root) => root.ValueKind == JsonValueKind.Object && root.TryGetProperty(\"model\", out var m) && m.ValueKind == JsonValueKind.Object;","tryCatchPattern":"try { var tok = SentencePieceTokenizer.CreateFromTokenizerJson(stream); } catch (InvalidDataException ex) { throw new InvalidOperationException(\"Invalid or wrong tokenizer.json file.\", ex); }","preventionTips":["Confirm the file is tokenizer.json, not tokenizer_config.json","Validate JSON shape before passing to the API","Checksum downloaded tokenizer files"],"tags":["schema-validation","tokenizer-json","huggingface"],"backgroundTag":"schema-validation-failed","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}