{"record":{"id":"4b3d6be80f47cfec","repo":"dotnet/machinelearning","slug":"the-tokenizer-json-model-unk-id-property-must-be","errorCode":null,"errorMessage":"The tokenizer.json model 'unk_id' property must be a number or null.","messagePattern":"The tokenizer\\.json model 'unk_id' property must be a number or null\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs","lineNumber":613,"sourceCode":"                    throw new InvalidDataException($\"Expected model type 'Unigram' but found '{modelTypeElement.GetString()}'.\");\n                }\n            }\n            else if (modelElement.TryGetProperty(\"merges\", out _))\n            {\n                throw new InvalidDataException(\"The tokenizer.json 'model' has no 'type' and contains 'merges'; this factory only supports 'Unigram' models.\");\n            }\n\n            if (!modelElement.TryGetProperty(\"unk_id\", out JsonElement unkIdElement))\n            {\n                throw new InvalidDataException(\"The tokenizer.json model does not contain an 'unk_id' property.\");\n            }\n\n            // HF permits a null unk_id, meaning the model has no unknown token; represent that as -1 and validate the\n            // byte_fallback pairing below (a number is validated against the vocabulary once it has been parsed).\n            bool unkIsNull = unkIdElement.ValueKind == JsonValueKind.Null;\n            if (!unkIsNull && unkIdElement.ValueKind != JsonValueKind.Number)\n            {\n                throw new InvalidDataException(\"The tokenizer.json model 'unk_id' property must be a number or null.\");\n            }\n\n            int unkId = unkIsNull ? -1 : unkIdElement.GetInt32();\n\n            bool byteFallback = modelElement.TryGetProperty(\"byte_fallback\", out JsonElement byteFallbackElement) &&\n                                byteFallbackElement.ValueKind == JsonValueKind.True;\n\n            if (!modelElement.TryGetProperty(\"vocab\", out JsonElement vocabElement) ||\n                vocabElement.ValueKind != JsonValueKind.Array)\n            {\n                throw new InvalidDataException(\"The tokenizer.json model does not contain a valid 'vocab' array.\");\n            }\n\n            List<(string Piece, float Score)> vocab = new List<(string Piece, float Score)>(vocabElement.GetArrayLength());\n            foreach (JsonElement entry in vocabElement.EnumerateArray())\n            {\n                if (entry.ValueKind != JsonValueKind.Array || entry.GetArrayLength() < 2)\n                {","sourceCodeStart":595,"sourceCodeEnd":631,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs#L595-L631","documentation":"The 'unk_id' property exists but is neither a JSON number nor null. The loader accepts only an integer vocabulary index or null (meaning 'no unknown token'); any other JSON kind (string, bool, object, array) is rejected with InvalidDataException.","triggerScenarios":"CreateFromTokenizerJson with model.unk_id set to e.g. \"0\" (a string), true, or an object/array.","commonSituations":"Hand-edited tokenizer.json quoting the id; JSON produced by tools that serialize ids as strings; copy-paste mistakes when constructing the model section manually.","solutions":["Change unk_id to an unquoted integer, e.g. \"unk_id\": 0.","Use null if the model has no unknown token, ensuring byte_fallback is true.","Validate the tokenizer.json with the Hugging Face tokenizers library before loading it here."],"exampleFix":"// before\n\"unk_id\": \"0\"\n// after\n\"unk_id\": 0","handlingStrategy":"validation","validationCode":"var unkId = model.GetProperty(\"unk_id\");\nif (unkId.ValueKind is not (JsonValueKind.Number or JsonValueKind.Null))\n    throw new InvalidOperationException(\"unk_id must be a number or null.\");","typeGuard":null,"tryCatchPattern":"try { return SentencePieceTokenizer.CreateFromTokenizerJson(stream); }\ncatch (InvalidDataException ex) when (ex.Message.Contains(\"must be a number or null\")) { /* repair or reject the tokenizer.json */ }","preventionTips":["Never quote numeric ids when generating tokenizer.json.","Deserialize-then-serialize the JSON with a typed model to normalize kinds."],"tags":["tokenizer","json-validation","type-mismatch"],"backgroundTag":"config-type-mismatch","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}