{"record":{"id":"866be973bfc6e049","repo":"dotnet/machinelearning","slug":"throw-new-argumentnullexception-nameof-mergestream","errorCode":null,"errorMessage":"throw new ArgumentNullException(nameof(mergeStream));","messagePattern":"throw new ArgumentNullException\\(nameof\\(mergeStream\\)\\);","errorType":"exception","errorClass":"ArgumentNullException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs","lineNumber":113,"sourceCode":"                bool addEndOfSentence = false,\n                string? unknownToken = DefaultSpecialToken,\n                string? beginningOfSentenceToken = DefaultSpecialToken,\n                string? endOfSentenceToken = DefaultSpecialToken) :\n            this(vocabularyStream, mergeStream, preTokenizer, normalizer, specialTokens, addPrefixSpace, addBeginningOfSentence, addEndOfSentence, unknownToken, beginningOfSentenceToken, endOfSentenceToken, disposeStream: false)\n        {\n        }\n\n        private CodeGenTokenizer(Stream vocabularyStream, Stream mergeStream, PreTokenizer? preTokenizer, Normalizer? normalizer, IReadOnlyDictionary<string, int>? specialTokens, bool addPrefixSpace,\n                        bool addBeginningOfSentence, bool addEndOfSentence, string? unknownToken, string? beginningOfSentenceToken, string? endOfSentenceToken, bool disposeStream)\n        {\n            if (vocabularyStream is null)\n            {\n                throw new ArgumentNullException(nameof(vocabularyStream));\n            }\n\n            if (mergeStream is null)\n            {\n                throw new ArgumentNullException(nameof(mergeStream));\n            }\n\n            _preTokenizer = preTokenizer;\n            _normalizer = normalizer;\n\n            // Tokenizer data files can be found in codegen-350M-mono\n            // https://huggingface.co/Salesforce/codegen-350M-mono/resolve/main/vocab.json?download=true\n            // https://huggingface.co/Salesforce/codegen-350M-mono/resolve/main/merges.txt?download=true\n\n            // Or in Phi-2\n            // https://huggingface.co/microsoft/phi-2/resolve/main/vocab.json?download=true\n            // https://huggingface.co/microsoft/phi-2/resolve/main/merges.txt?download=true\n\n            _vocab = GetVocabulary(vocabularyStream);\n            _vocabReverse = _vocab.ToDictionary(kvp => kvp.Value.Id, kvp => kvp.Value.Token);\n            _mergeRanks = GetMergeRanks(mergeStream);\n            _cache = new StringSpanOrdinalKeyCache<List<EncodedToken>>();\n","sourceCodeStart":95,"sourceCodeEnd":131,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs#L95-L131","documentation":"The private CodeGenTokenizer constructor throws ArgumentNullException when mergeStream is null. Both the vocabulary stream and the merges stream must be supplied since the constructor reads the BPE merge rules from them. The check follows the vocabularyStream check, so both arguments must be non-null streams.","triggerScenarios":"new CodeGenTokenizer(vocabStream, null) via the public stream constructor; a helper that opens the vocab file succeeded but the merges File.OpenRead was skipped or its result not assigned.","commonSituations":"Resource-loading code that opens two streams returns null for the second due to a missing embedded resource; refactoring dropped the merges stream initialization.","solutions":["Pass a non-null opened Stream for merges.txt as the second stream argument.","Ensure both streams are opened together, e.g. File.OpenRead for vocab and merges in the same factory method.","Verify embedded resource names for the merges asset if loading from an assembly."],"exampleFix":"// before\nvar tokenizer = new CodeGenTokenizer(File.OpenRead(vocabPath), mergesStream); // mergesStream was never assigned\n// after\nusing var vocabStream = File.OpenRead(vocabPath);\nusing var mergesStream = File.OpenRead(mergesPath);\nvar tokenizer = new CodeGenTokenizer(vocabStream, mergesStream);","handlingStrategy":"type-guard","validationCode":"using var vocabStream = File.OpenRead(vocabPath);\nusing var mergeStream = File.OpenRead(mergesPath);\nif (vocabStream is null || mergeStream is null) throw new InvalidOperationException(\"Tokenizer streams must be opened\");","typeGuard":"if (mergeStream is Stream ms) { /* construct with ms */ }","tryCatchPattern":"try { var tok = new CodeGenTokenizer(vocabStream, mergeStream); } catch (ArgumentNullException ex) { logger.LogError(ex, \"Null stream argument: {Param}\", ex.ParamName); }","preventionTips":["Open both vocab and merges streams in the same factory function","Dispose streams in a using scope around tokenizer usage","Fail loudly when a stream opener returns null"],"tags":["csharp","argument-null","stream"],"backgroundTag":"null-argument","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}