{"record":{"id":"23fbcca82bb588da","repo":"dotnet/machinelearning","slug":"expected-model-type-unigram-but-found-modeltyp","errorCode":null,"errorMessage":"Expected model type 'Unigram' but found '{modelTypeElement.GetString()}'.","messagePattern":"Expected model type 'Unigram' but found '(.+?)'\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs","lineNumber":595,"sourceCode":"            if (!root.TryGetProperty(\"model\", out JsonElement modelElement))\n            {\n                throw new InvalidDataException(\"The tokenizer.json does not contain a 'model' property.\");\n            }\n\n            if (modelElement.ValueKind != JsonValueKind.Object)\n            {\n                throw new InvalidDataException(\"The tokenizer.json 'model' property must be a JSON object.\");\n            }\n\n            // Validate the model is Unigram. Older tokenizer.json files (e.g. xlm-roberta-base, albert) omit the\n            // model \"type\" entirely; treat a model that has a \"vocab\" but no BPE \"merges\" as Unigram, which matches\n            // how the Hugging Face loaders disambiguate these files.\n            if (modelElement.TryGetProperty(\"type\", out JsonElement modelTypeElement) &&\n                modelTypeElement.ValueKind == JsonValueKind.String)\n            {\n                if (!string.Equals(modelTypeElement.GetString(), \"Unigram\", StringComparison.OrdinalIgnoreCase))\n                {\n                    throw new InvalidDataException($\"Expected model type 'Unigram' but found '{modelTypeElement.GetString()}'.\");\n                }\n            }\n            else if (modelElement.TryGetProperty(\"merges\", out _))\n            {\n                throw new InvalidDataException(\"The tokenizer.json 'model' has no 'type' and contains 'merges'; this factory only supports 'Unigram' models.\");\n            }\n\n            if (!modelElement.TryGetProperty(\"unk_id\", out JsonElement unkIdElement))\n            {\n                throw new InvalidDataException(\"The tokenizer.json model does not contain an 'unk_id' property.\");\n            }\n\n            // HF permits a null unk_id, meaning the model has no unknown token; represent that as -1 and validate the\n            // byte_fallback pairing below (a number is validated against the vocabulary once it has been parsed).\n            bool unkIsNull = unkIdElement.ValueKind == JsonValueKind.Null;\n            if (!unkIsNull && unkIdElement.ValueKind != JsonValueKind.Number)\n            {\n                throw new InvalidDataException(\"The tokenizer.json model 'unk_id' property must be a number or null.\");","sourceCodeStart":577,"sourceCodeEnd":613,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs#L577-L613","documentation":"CreateFromTokenizerJson only supports Hugging Face Unigram (SentencePiece-style) models. When tokenizer.json's 'model' object has a 'type' field whose value is not 'Unigram' (case-insensitive), this InvalidDataException is thrown to reject BPE/WordPiece/WordLevel and other model types that this factory cannot load.","triggerScenarios":"Calling SentencePieceTokenizer.CreateFromTokenizerJson (or the HF tokenizer.json loader path) with a tokenizer.json whose model.type is 'BPE', 'WordPiece', 'WordLevel', 'FastBPE', or any misspelled/alternate casing variant other than 'Unigram'.","commonSituations":"Pointing the factory at a tokenizer.json exported from a GPT-2/Llama-BPE or BERT WordPiece tokenizer instead of a SentencePiece/Unigram model (e.g. Llama 2, T5, XLNet are Unigram; GPT-2/Phi are BPE); typos in the type field; hand-edited tokenizer.json files.","solutions":["Use the correct tokenizer factory for the model type (e.g. the BPE/WordPiece tokenizer classes in Microsoft.ML.Tokenizers) instead of SentencePieceTokenizer.","Obtain a tokenizer.json produced from a genuine SentencePiece Unigram model (e.g. Llama, T5) rather than a BPE/WordPiece one.","Fix the model.type field if it is a typo; it must be exactly 'Unigram' (case-insensitive)."],"exampleFix":"// before\n\"model\": { \"type\": \"BPE\", \"vocab\": ..., \"merges\": ... }\n// after\n\"model\": { \"type\": \"Unigram\", \"unk_id\": 0, \"vocab\": [[\"<unk>\", 0.0], ...] }","handlingStrategy":"validation","validationCode":"using var doc = JsonDocument.Parse(tokenizerJson);\nvar model = doc.RootElement.GetProperty(\"model\");\nif (model.TryGetProperty(\"type\", out var t) &&\n    !string.Equals(t.GetString(), \"Unigram\", StringComparison.OrdinalIgnoreCase))\n    throw new InvalidOperationException($\"Model type '{t.GetString()}' not supported by SentencePieceTokenizer.\");","typeGuard":null,"tryCatchPattern":"try { var tok = SentencePieceTokenizer.CreateFromTokenizerJson(stream); }\ncatch (InvalidDataException ex) { log.LogError(ex, \"Unsupported tokenizer model type\"); throw new UnsupportedTokenizerException(...); }","preventionTips":["Confirm the source model is SentencePiece/Unigram (Llama 2, T5, XLNet) before loading.","Check model.type in tokenizer.json during a CI validation step.","Use model-type-specific tokenizer factories for BPE/WordPiece tokenizers."],"tags":["tokenizer","json-validation","unsupported-model-type"],"backgroundTag":"incompatible-source-type","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}