{"record":{"id":"2f784c62555322c4","repo":"dotnet/machinelearning","slug":"the-tokenizer-json-model-does-not-contain-an-unk","errorCode":null,"errorMessage":"The tokenizer.json model does not contain an 'unk_id' property.","messagePattern":"The tokenizer\\.json model does not contain an 'unk_id' property\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs","lineNumber":605,"sourceCode":"            // Validate the model is Unigram. Older tokenizer.json files (e.g. xlm-roberta-base, albert) omit the\n            // model \"type\" entirely; treat a model that has a \"vocab\" but no BPE \"merges\" as Unigram, which matches\n            // how the Hugging Face loaders disambiguate these files.\n            if (modelElement.TryGetProperty(\"type\", out JsonElement modelTypeElement) &&\n                modelTypeElement.ValueKind == JsonValueKind.String)\n            {\n                if (!string.Equals(modelTypeElement.GetString(), \"Unigram\", StringComparison.OrdinalIgnoreCase))\n                {\n                    throw new InvalidDataException($\"Expected model type 'Unigram' but found '{modelTypeElement.GetString()}'.\");\n                }\n            }\n            else if (modelElement.TryGetProperty(\"merges\", out _))\n            {\n                throw new InvalidDataException(\"The tokenizer.json 'model' has no 'type' and contains 'merges'; this factory only supports 'Unigram' models.\");\n            }\n\n            if (!modelElement.TryGetProperty(\"unk_id\", out JsonElement unkIdElement))\n            {\n                throw new InvalidDataException(\"The tokenizer.json model does not contain an 'unk_id' property.\");\n            }\n\n            // HF permits a null unk_id, meaning the model has no unknown token; represent that as -1 and validate the\n            // byte_fallback pairing below (a number is validated against the vocabulary once it has been parsed).\n            bool unkIsNull = unkIdElement.ValueKind == JsonValueKind.Null;\n            if (!unkIsNull && unkIdElement.ValueKind != JsonValueKind.Number)\n            {\n                throw new InvalidDataException(\"The tokenizer.json model 'unk_id' property must be a number or null.\");\n            }\n\n            int unkId = unkIsNull ? -1 : unkIdElement.GetInt32();\n\n            bool byteFallback = modelElement.TryGetProperty(\"byte_fallback\", out JsonElement byteFallbackElement) &&\n                                byteFallbackElement.ValueKind == JsonValueKind.True;\n\n            if (!modelElement.TryGetProperty(\"vocab\", out JsonElement vocabElement) ||\n                vocabElement.ValueKind != JsonValueKind.Array)\n            {","sourceCodeStart":587,"sourceCodeEnd":623,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs#L587-L623","documentation":"A Unigram model requires an 'unk_id' property (the vocabulary id of the unknown-token piece, or explicit null). The loader refuses to guess it, so a tokenizer.json whose model lacks this field is rejected with InvalidDataException.","triggerScenarios":"CreateFromTokenizerJson with a model object that has type 'Unigram' and a 'vocab' but no 'unk_id' key.","commonSituations":"Hand-written or minimal tokenizer.json files; exports from custom SentencePiece training where the id was stripped; copying only the vocab from a larger file.","solutions":["Add \"unk_id\": <int> to the model object, set to the index of the '<unk>' piece in vocab.","Set \"unk_id\": null if the model genuinely has no unknown token (requires byte_fallback enabled).","Re-export the tokenizer with the tokenizers library so all required Unigram fields are present."],"exampleFix":"// before\n\"model\": { \"type\": \"Unigram\", \"vocab\": [[\"<unk>\", 0.0], ...] }\n// after\n\"model\": { \"type\": \"Unigram\", \"unk_id\": 0, \"vocab\": [[\"<unk>\", 0.0], ...] }","handlingStrategy":"validation","validationCode":"var model = JsonDocument.Parse(tokenizerJson).RootElement.GetProperty(\"model\");\nif (!model.TryGetProperty(\"unk_id\", out _))\n    throw new InvalidOperationException(\"model.unk_id is required (number or null).\");","typeGuard":null,"tryCatchPattern":"try { var tok = SentencePieceTokenizer.CreateFromTokenizerJson(stream); }\ncatch (InvalidDataException ex) when (ex.Message.Contains(\"unk_id\")) { log.LogError(ex, \"tokenizer.json missing unk_id\"); throw; }","preventionTips":["Keep unk_id present in every tokenizer.json you ship.","Validate tokenizer.json with the HF tokenizers library as a pre-flight check."],"tags":["tokenizer","json-validation","missing-field"],"backgroundTag":"missing-required-config-field","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}