{"record":{"id":"d57d7f5c051dc9af","repo":"dotnet/machinelearning","slug":"a-piece-string-in-model-vocab-is-null","errorCode":null,"errorMessage":"A piece string in 'model.vocab' is null.","messagePattern":"A piece string in 'model\\.vocab' is null\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs","lineNumber":643,"sourceCode":"            }\n\n            List<(string Piece, float Score)> vocab = new List<(string Piece, float Score)>(vocabElement.GetArrayLength());\n            foreach (JsonElement entry in vocabElement.EnumerateArray())\n            {\n                if (entry.ValueKind != JsonValueKind.Array || entry.GetArrayLength() < 2)\n                {\n                    throw new InvalidDataException(\"Each entry in 'model.vocab' must be a [piece, score] array.\");\n                }\n\n                if (entry[0].ValueKind != JsonValueKind.String || entry[1].ValueKind != JsonValueKind.Number)\n                {\n                    throw new InvalidDataException(\"Each entry in 'model.vocab' must be a [string piece, number score] pair.\");\n                }\n\n                string? piece = entry[0].GetString();\n                if (piece is null)\n                {\n                    throw new InvalidDataException(\"A piece string in 'model.vocab' is null.\");\n                }\n\n                vocab.Add((piece, entry[1].GetSingle()));\n            }\n\n            if (unkIsNull)\n            {\n                // Without an unknown token the only way to represent out-of-vocabulary input is byte fallback; a model\n                // with neither cannot encode OOV text, so reject that combination up front rather than emitting an\n                // invalid token id at encode time.\n                if (!byteFallback)\n                {\n                    throw new NotSupportedException(\"The tokenizer.json model has a null 'unk_id' but does not enable 'byte_fallback'; a Unigram model without an unknown token is only supported when byte_fallback is enabled.\");\n                }\n            }\n            else if (unkId < 0 || unkId >= vocab.Count)\n            {\n                throw new InvalidDataException($\"The tokenizer.json model 'unk_id' ({unkId}) is out of range for a vocabulary of {vocab.Count} pieces.\");","sourceCodeStart":625,"sourceCodeEnd":661,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs#L625-L661","documentation":"A defensive check after GetString(): although the ValueKind check should have caught it, a piece that deserializes to a null string is rejected with InvalidDataException to prevent null pieces entering the vocabulary.","triggerScenarios":"CreateFromTokenizerJson with a vocab entry whose piece element is JSON null or a value that GetString() yields null for.","commonSituations":"Programmatically generated tokenizer.json with missing piece values; corrupted or partially written vocab arrays.","solutions":["Replace the null piece with a valid non-empty piece string.","Regenerate tokenizer.json from the original model so all pieces are populated.","Pre-validate that every vocab entry has a non-null string in slot 0."],"exampleFix":"// before\n\"vocab\": [[null, 0.0]]\n// after\n\"vocab\": [[\"<unk>\", 0.0]]","handlingStrategy":"validation","validationCode":"foreach (var e in model.GetProperty(\"vocab\").EnumerateArray())\n    if (e[0].ValueKind != JsonValueKind.String || e[0].GetString() is null)\n        throw new InvalidOperationException(\"Vocab piece must be a non-null string.\");","typeGuard":null,"tryCatchPattern":"try { return SentencePieceTokenizer.CreateFromTokenizerJson(stream); }\ncatch (InvalidDataException ex) when (ex.Message.Contains(\"piece string\")) { /* regenerate tokenizer.json */ }","preventionTips":["Avoid hand-generating vocab arrays; export from the trained model.","Check for null/missing fields after any JSON manipulation."],"tags":["tokenizer","json-validation","null-value"],"backgroundTag":"null-argument","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}