{"record":{"id":"8af7e6ec6fc260c9","repo":"dotnet/machinelearning","slug":"vocab","errorCode":null,"errorMessage":"vocab","messagePattern":"vocab","errorType":"exception","errorClass":"ArgumentNullException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceUnigramModel.cs","lineNumber":308,"sourceCode":"            }\n        }\n\n        private static void AddControlId(HashSet<int> set, int id)\n        {\n            if (id >= 0)\n            {\n                set.Add(id);\n            }\n        }\n\n        private static int GetPieceCount(IReadOnlyList<(string Piece, float Score)>? pieces)\n            => pieces?.Count ?? 0;\n\n        private static string GetPieceAtIndex(IReadOnlyList<(string Piece, float Score)>? pieces, int index)\n        {\n            if (pieces is null)\n            {\n                throw new ArgumentNullException(\"vocab\");\n            }\n\n            // A negative index means the model has no unknown token (HF permits a null unk_id). Return a cosmetic\n            // default token that is never emitted (OOV is handled by byte fallback in that configuration).\n            if (index < 0)\n            {\n                return \"<unk>\";\n            }\n\n            if (index >= pieces.Count)\n            {\n                throw new ArgumentOutOfRangeException(\"unkId\", \"unkId must be a valid index in the vocabulary.\");\n            }\n\n            return pieces[index].Piece;\n        }\n\n        // Validates pieces is not null and unkId is in range; returns pieces unchanged.","sourceCodeStart":290,"sourceCodeEnd":326,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceUnigramModel.cs#L290-L326","documentation":"GetPieceAtIndex requires a non-null vocabulary list; when it is called with a null pieces list it throws ArgumentNullException with the parameter name \"vocab\". This is an internal guard reached when a SentencePieceUnigramModel was constructed without a valid vocabulary.","triggerScenarios":"Internal lookups (e.g. via SentencePieceUnigramModel special-token accessors) calling GetPieceAtIndex with pieces == null, which reflects a model instance created with a null vocabulary.","commonSituations":"Programmatically constructing the unigram model with a null/missing vocabulary collection and later querying token names; deserialization paths that left the vocab unset.","solutions":["Ensure the vocabulary passed to the unigram model constructor is a non-null list of (Piece, Score) tuples.","Call ValidateVocab-style checks (pieces != null, unkId in range) before using the model.","Reload the model from a valid tokenizer.json/proto so the vocabulary is populated."],"exampleFix":"// before\nmodel = new SentencePieceUnigramModel(pieces: null, unkId: 0);\n// after\nmodel = new SentencePieceUnigramModel(pieces: loadedPieces, unkId: 0); // loadedPieces != null","handlingStrategy":"type-guard","validationCode":"// C# — ensure vocab is non-null before constructing/querying the model\nif (pieces is null)\n    throw new InvalidOperationException(\"Vocabulary must be loaded before creating SentencePieceUnigramModel.\");","typeGuard":"static bool HasVocab(IReadOnlyList<(string, float)>? p) => p is { Count: > 0 };","tryCatchPattern":"try { var piece = GetPieceAt(model, id); }\ncatch (ArgumentNullException ex) when (ex.ParamName == \"vocab\") { /* vocab was null — reload model */ }","preventionTips":["Always load the vocabulary before model construction.","Fail fast on null vocab at deserialization boundaries.","Avoid constructing models from partially parsed tokenizer.json files.","Use ValidateVocab-equivalent checks as the first step of any custom loader."],"tags":["sentencepiece","null-argument","vocabulary","internal"],"backgroundTag":"null-argument","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}