{"record":{"id":"ce255b049e557916","repo":"dotnet/machinelearning","slug":"the-vocabulary-cannot-be-null","errorCode":null,"errorMessage":"The vocabulary cannot be null.","messagePattern":"The vocabulary cannot be null\\.","errorType":"exception","errorClass":"ArgumentNullException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs","lineNumber":149,"sourceCode":"\n            return new BpeTokenizer(result.vocab, result.merges, preTokenizer, normalizer, specialTokens, unknownToken, continuingSubwordPrefix, endOfWordSuffix, fuseUnknownTokens);\n        }\n\n        /// <summary>\n        /// Create a new Bpe tokenizer object to use for text encoding.\n        /// </summary>\n        /// <param name=\"options\">The options used to create the Bpe tokenizer.</param>\n        /// <returns>The Bpe tokenizer object.</returns>\n        public static BpeTokenizer Create(BpeOptions options)\n        {\n            if (options is null)\n            {\n                throw new ArgumentNullException(nameof(options));\n            }\n\n            if (options.Vocabulary is null)\n            {\n                throw new ArgumentNullException(nameof(options.Vocabulary), \"The vocabulary cannot be null.\");\n            }\n\n            Dictionary<StringSpanOrdinalKey, int> vocab = new Dictionary<StringSpanOrdinalKey, int>(1000);\n\n            foreach (KeyValuePair<string, int> kvp in options.Vocabulary)\n            {\n                vocab.Add(new StringSpanOrdinalKey(kvp.Key), kvp.Value);\n            }\n\n            if (vocab.Count == 0)\n            {\n                throw new InvalidOperationException(\"The vocabulary cannot be empty.\");\n            }\n\n            Vec<(string, string)> merges = default;\n            if (options.Merges is not null)\n            {\n                merges = new Vec<(string, string)>(1000);","sourceCodeStart":131,"sourceCodeEnd":167,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs#L131-L167","documentation":"BpeTokenizer.Create(BpeOptions) requires options.Vocabulary to be a non-null dictionary mapping tokens to ids; a null Vocabulary throws ArgumentNullException with message 'The vocabulary cannot be null.' The BPE model cannot encode/decode without token-id assignments.","triggerScenarios":"Passing a BpeOptions whose Vocabulary property was never assigned (default null) to BpeTokenizer.Create — e.g. setting only Merges or file paths on the options object.","commonSituations":"Assuming Vocabulary can be loaded lazily or from VocabularyFile via the options object (it cannot in this overload); partial initialization after refactoring; deserialization leaving Vocabulary null.","solutions":["Populate options.Vocabulary with a Dictionary<string,int> of token->id before calling Create.","If loading from a file, use the file-based Create(vocabFile, mergesFile) overload instead.","Check that deserialization/config binding actually fills Vocabulary (key names, case sensitivity).","Guard: if (options?.Vocabulary == null) load or throw with a clear message before Create."],"exampleFix":"// before\nvar options = new BpeOptions { Merges = mergesList }; // Vocabulary null\nvar tokenizer = BpeTokenizer.Create(options);\n// after\nvar options = new BpeOptions { Merges = mergesList, Vocabulary = vocabDict };\nvar tokenizer = BpeTokenizer.Create(options);","handlingStrategy":"validation","validationCode":"if (options?.Vocabulary == null || options.Vocabulary.Count == 0)\n    throw new InvalidOperationException(\"BpeOptions.Vocabulary must be non-null and non-empty.\");","typeGuard":"static bool HasVocabulary(BpeOptions? o) => o?.Vocabulary is { Count: > 0 };","tryCatchPattern":"try { var t = BpeTokenizer.Create(options); }\ncatch (ArgumentNullException ex) when (ex.Message.Contains(\"vocabulary cannot be null\"))\n{ options.Vocabulary = LoadVocabFromDisk(); var t = BpeTokenizer.Create(options); }","preventionTips":["Always assign Vocabulary (and Merges if used) when building BpeOptions.","Use the file-based Create overload when the vocab lives on disk.","Validate options completeness in a single builder method."],"tags":["dotnet","tokenizers","bpe","vocabulary"],"backgroundTag":"null-argument","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}