{"record":{"id":"482d886220ed417a","repo":"dotnet/machinelearning","slug":"the-tokenizer-json-model-has-no-type-and-conta","errorCode":null,"errorMessage":"The tokenizer.json 'model' has no 'type' and contains 'merges'; this factory only supports 'Unigram' models.","messagePattern":"The tokenizer\\.json 'model' has no 'type' and contains 'merges'; this factory only supports 'Unigram' models\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs","lineNumber":600,"sourceCode":"            if (modelElement.ValueKind != JsonValueKind.Object)\n            {\n                throw new InvalidDataException(\"The tokenizer.json 'model' property must be a JSON object.\");\n            }\n\n            // Validate the model is Unigram. Older tokenizer.json files (e.g. xlm-roberta-base, albert) omit the\n            // model \"type\" entirely; treat a model that has a \"vocab\" but no BPE \"merges\" as Unigram, which matches\n            // how the Hugging Face loaders disambiguate these files.\n            if (modelElement.TryGetProperty(\"type\", out JsonElement modelTypeElement) &&\n                modelTypeElement.ValueKind == JsonValueKind.String)\n            {\n                if (!string.Equals(modelTypeElement.GetString(), \"Unigram\", StringComparison.OrdinalIgnoreCase))\n                {\n                    throw new InvalidDataException($\"Expected model type 'Unigram' but found '{modelTypeElement.GetString()}'.\");\n                }\n            }\n            else if (modelElement.TryGetProperty(\"merges\", out _))\n            {\n                throw new InvalidDataException(\"The tokenizer.json 'model' has no 'type' and contains 'merges'; this factory only supports 'Unigram' models.\");\n            }\n\n            if (!modelElement.TryGetProperty(\"unk_id\", out JsonElement unkIdElement))\n            {\n                throw new InvalidDataException(\"The tokenizer.json model does not contain an 'unk_id' property.\");\n            }\n\n            // HF permits a null unk_id, meaning the model has no unknown token; represent that as -1 and validate the\n            // byte_fallback pairing below (a number is validated against the vocabulary once it has been parsed).\n            bool unkIsNull = unkIdElement.ValueKind == JsonValueKind.Null;\n            if (!unkIsNull && unkIdElement.ValueKind != JsonValueKind.Number)\n            {\n                throw new InvalidDataException(\"The tokenizer.json model 'unk_id' property must be a number or null.\");\n            }\n\n            int unkId = unkIsNull ? -1 : unkIdElement.GetInt32();\n\n            bool byteFallback = modelElement.TryGetProperty(\"byte_fallback\", out JsonElement byteFallbackElement) &&","sourceCodeStart":582,"sourceCodeEnd":618,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs#L582-L618","documentation":"If the 'model' object has no 'type' field but does contain a 'merges' array, the loader infers it is a BPE-style model, which this factory does not support. The throw makes the inference explicit instead of failing later with a confusing vocabulary error.","triggerScenarios":"CreateFromTokenizerJson given a tokenizer.json whose model object omits 'type' but includes 'merges' (the BPE signature).","commonSituations":"Older or hand-trimmed tokenizer.json exports that dropped the type field; GPT-2-style BPE tokenizers; minimal tokenizer.json snippets copied from blog posts.","solutions":["Add \"type\": \"Unigram\" only if the model truly is Unigram; otherwise use the BPE tokenizer loader instead.","Regenerate tokenizer.json with a complete model section (convert_tokenizer from tokenizers lib preserves the type field).","Switch to a tokenizer.json for a SentencePiece/Unigram model."],"exampleFix":"// before\n\"model\": { \"vocab\": [...], \"merges\": [...] }\n// after (if truly Unigram)\n\"model\": { \"type\": \"Unigram\", \"unk_id\": 0, \"vocab\": [...] }","handlingStrategy":"validation","validationCode":"var model = JsonDocument.Parse(tokenizerJson).RootElement.GetProperty(\"model\");\nif (!model.TryGetProperty(\"type\", out _) && model.TryGetProperty(\"merges\", out _))\n    throw new InvalidOperationException(\"tokenizer.json is BPE-style; use a BPE loader.\");","typeGuard":null,"tryCatchPattern":"try { return SentencePieceTokenizer.CreateFromTokenizerJson(stream); }\ncatch (InvalidDataException ex) when (ex.Message.Contains(\"'merges'\")) { /* fall back to BPE tokenizer loader */ }","preventionTips":["Always export tokenizer.json with the full model section including 'type'.","Lint tokenizer.json files for required fields before deployment."],"tags":["tokenizer","json-validation","missing-type-field"],"backgroundTag":"incompatible-source-type","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}