{"record":{"id":"c63ca1fd4030abe7","repo":"dotnet/machinelearning","slug":"throw-new-argumentnullexception-nameof-vocabularys","errorCode":null,"errorMessage":"throw new ArgumentNullException(nameof(vocabularyStream));","messagePattern":"throw new ArgumentNullException\\(nameof\\(vocabularyStream\\)\\);","errorType":"exception","errorClass":"ArgumentNullException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs","lineNumber":108,"sourceCode":"                PreTokenizer? preTokenizer = null,\n                Normalizer? normalizer = null,\n                IReadOnlyDictionary<string, int>? specialTokens = null,\n                bool addPrefixSpace = false,\n                bool addBeginningOfSentence = false,\n                bool addEndOfSentence = false,\n                string? unknownToken = DefaultSpecialToken,\n                string? beginningOfSentenceToken = DefaultSpecialToken,\n                string? endOfSentenceToken = DefaultSpecialToken) :\n            this(vocabularyStream, mergeStream, preTokenizer, normalizer, specialTokens, addPrefixSpace, addBeginningOfSentence, addEndOfSentence, unknownToken, beginningOfSentenceToken, endOfSentenceToken, disposeStream: false)\n        {\n        }\n\n        private CodeGenTokenizer(Stream vocabularyStream, Stream mergeStream, PreTokenizer? preTokenizer, Normalizer? normalizer, IReadOnlyDictionary<string, int>? specialTokens, bool addPrefixSpace,\n                        bool addBeginningOfSentence, bool addEndOfSentence, string? unknownToken, string? beginningOfSentenceToken, string? endOfSentenceToken, bool disposeStream)\n        {\n            if (vocabularyStream is null)\n            {\n                throw new ArgumentNullException(nameof(vocabularyStream));\n            }\n\n            if (mergeStream is null)\n            {\n                throw new ArgumentNullException(nameof(mergeStream));\n            }\n\n            _preTokenizer = preTokenizer;\n            _normalizer = normalizer;\n\n            // Tokenizer data files can be found in codegen-350M-mono\n            // https://huggingface.co/Salesforce/codegen-350M-mono/resolve/main/vocab.json?download=true\n            // https://huggingface.co/Salesforce/codegen-350M-mono/resolve/main/merges.txt?download=true\n\n            // Or in Phi-2\n            // https://huggingface.co/microsoft/phi-2/resolve/main/vocab.json?download=true\n            // https://huggingface.co/microsoft/phi-2/resolve/main/merges.txt?download=true\n","sourceCodeStart":90,"sourceCodeEnd":126,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs#L90-L126","documentation":"The private CodeGenTokenizer constructor (used by the public path and stream overloads) throws ArgumentNullException when vocabularyStream is null. All stream data (vocab, merges) is read from these streams, so a null vocab stream cannot be tolerated. Callers hitting this directly usually invoked the stream-based public constructor with null.","triggerScenarios":"new CodeGenTokenizer((Stream)null!, mergesStream) or a factory method returning null for the vocab stream, e.g. Assembly.GetManifestResourceStream returning null because the embedded resource name is wrong.","commonSituations":"Embedded resource typo (namespace/name mismatch) makes GetManifestResourceStream return null; conditional asset loading returned null silently.","solutions":["Pass a non-null, opened Stream for the vocabulary (e.g. File.OpenRead(vocabPath)).","If loading an embedded resource, verify the resource name with assembly.GetManifestResourceNames() before calling GetManifestResourceStream.","Check the stream-opening call for a null return instead of assuming it always succeeds."],"exampleFix":"// before\nvar vocab = assembly.GetManifestResourceStream(\"Wrong.Name.vocab.json\"); // null\nvar tokenizer = new CodeGenTokenizer(vocab, mergesStream);\n// after\nvar vocab = assembly.GetManifestResourceStream(\"MyApp.Models.vocab.json\") ?? throw new InvalidOperationException(\"Embedded vocab resource not found\");\nvar tokenizer = new CodeGenTokenizer(vocab, mergesStream);","handlingStrategy":"type-guard","validationCode":"var vocabStream = assembly.GetManifestResourceStream(resourceName);\nif (vocabStream is null) throw new InvalidOperationException($\"Embedded resource '{resourceName}' not found\");","typeGuard":"if (vocabStream is Stream vs) { /* construct with vs */ }","tryCatchPattern":"try { var tok = new CodeGenTokenizer(vocabStream, mergeStream); } catch (ArgumentNullException ex) { logger.LogError(ex, \"Null stream argument: {Param}\", ex.ParamName); }","preventionTips":["Check GetManifestResourceStream return for null before use","List actual resource names via GetManifestResourceNames during setup","Wrap stream loading in a helper that never returns null"],"tags":["csharp","argument-null","stream"],"backgroundTag":"null-argument","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}