{"record":{"id":"4804a7147c97ac58","repo":"dotnet/machinelearning","slug":"throw-new-argumentnullexception-nameof-vocabstream","errorCode":null,"errorMessage":"throw new ArgumentNullException(nameof(vocabStream));","messagePattern":"throw new ArgumentNullException\\(nameof\\(vocabStream\\)\\);","errorType":"exception","errorClass":"ArgumentNullException","httpStatus":null,"severity":"critical","filePath":"src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs","lineNumber":239,"sourceCode":"        /// <param name=\"endOfWordSuffix\">The suffix to attach to sub-word units that represent an end of word.</param>\n        /// <param name=\"fuseUnknownTokens\">Indicate whether allowing multiple unknown tokens get fused.</param>\n        /// <remarks>\n        /// When creating the tokenizer, ensure that the vocabulary stream is sourced from a trusted provider.\n        /// </remarks>\n        public static BpeTokenizer Create(\n                                Stream vocabStream,\n                                Stream? mergesStream,\n                                PreTokenizer? preTokenizer = null,\n                                Normalizer? normalizer = null,\n                                IReadOnlyDictionary<string, int>? specialTokens = null,\n                                string? unknownToken = null,\n                                string? continuingSubwordPrefix = null,\n                                string? endOfWordSuffix = null,\n                                bool fuseUnknownTokens = false)\n        {\n            if (vocabStream is null)\n            {\n                throw new ArgumentNullException(nameof(vocabStream));\n            }\n\n            (Dictionary<StringSpanOrdinalKey, int>? vocab, Vec<(string, string)> merges) result = ReadModelDataAsync(vocabStream, mergesStream, useAsync: false).GetAwaiter().GetResult();\n\n            return new BpeTokenizer(result.vocab, result.merges, preTokenizer, normalizer, specialTokens, unknownToken, continuingSubwordPrefix, endOfWordSuffix, fuseUnknownTokens);\n        }\n\n        /// <summary>\n        /// Create a new Bpe tokenizer object asynchronously to use for text encoding.\n        /// </summary>\n        /// <param name=\"vocabStream\">The JSON stream containing the dictionary of string keys and their ids.</param>\n        /// <param name=\"mergesStream\">The stream containing the tokens's pairs list.</param>\n        /// <param name=\"preTokenizer\">The pre-tokenizer to use.</param>\n        /// <param name=\"normalizer\">The normalizer to use.</param>\n        /// <param name=\"specialTokens\">The dictionary mapping special tokens to Ids.</param>\n        /// <param name=\"unknownToken\"> The unknown token to be used by the model.</param>\n        /// <param name=\"continuingSubwordPrefix\">The prefix to attach to sub-word units that don’t represent a beginning of word.</param>\n        /// <param name=\"endOfWordSuffix\">The suffix to attach to sub-word units that represent an end of word.</param>","sourceCodeStart":221,"sourceCodeEnd":257,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs#L221-L257","documentation":"BPETokenizer.Create requires a non-null vocabulary stream because the BPE model cannot be constructed without loading vocab data from it. The library throws ArgumentNullException synchronously at the start of the Create method to fail fast on a missing required argument. Passing null means the tokenizer would have no token-to-id mapping at all.","triggerScenarios":"Calling BpeTokenizer.Create(null, mergesStream, ...) or building the vocabStream argument from an expression (e.g. File.OpenRead on a null-configured path wrapped in a helper) that evaluates to null at runtime.","commonSituations":"Loading the vocab file path from configuration/environment variables that are unset; a helper method returning Stream? that returns null when the file is missing; refactoring where the vocab stream was moved to an optional parameter; deserializing model settings where the vocab file entry is absent.","solutions":["Ensure the vocabulary stream is non-null before calling Create (open the file or embed the resource).","Check that the config/env value holding the vocab file path is set and the file exists before opening the stream.","If you intend async loading, use CreateAsync with the same non-null guarantee.","Wrap stream creation so a missing file throws a clear FileNotFoundException instead of surfacing as a null stream later."],"exampleFix":"// before\nusing var vocab = File.OpenRead(config.VocabPath); // VocabPath may be null\nvar tok = BpeTokenizer.Create(vocab, mergesStream);\n// after\nif (string.IsNullOrEmpty(config.VocabPath)) throw new InvalidOperationException(\"Vocab path not configured\");\nusing var vocab = File.OpenRead(config.VocabPath);\nvar tok = BpeTokenizer.Create(vocab ?? throw new InvalidOperationException(\"vocab stream missing\"), mergesStream);","handlingStrategy":"validation","validationCode":"if (vocabStream is null) throw new InvalidOperationException(\"Vocabulary stream must be provided before calling BpeTokenizer.Create\");\nvar tokenizer = BpeTokenizer.Create(vocabStream, mergesStream);","typeGuard":"static bool HasVocabStream(Stream? s) => s is { CanRead: true };","tryCatchPattern":"try { var tok = BpeTokenizer.Create(vocabStream, mergesStream); }\ncatch (ArgumentNullException ex) when (ex.ParamName == \"vocabStream\") { throw new InvalidOperationException(\"BPE vocab stream was null; check model file loading\", ex); }","preventionTips":["Open model files with File.OpenRead and assign immediately — never keep Stream? fields that can stay null.","Fail at startup if the configured vocab path is missing.","Centralize tokenizer construction in one factory that validates all inputs."],"tags":["argument-null","tokenizer","bpe","csharp"],"backgroundTag":"null-argument","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}