dotnet/machinelearning · error · ArgumentNullException

throw new ArgumentNullException(nameof(vocabularyStream));

Error message

throw new ArgumentNullException(nameof(vocabularyStream));

What it means

The private CodeGenTokenizer constructor (used by the public path and stream overloads) throws ArgumentNullException when vocabularyStream is null. All stream data (vocab, merges) is read from these streams, so a null vocab stream cannot be tolerated. Callers hitting this directly usually invoked the stream-based public constructor with null.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs:108

                PreTokenizer? preTokenizer = null,
                Normalizer? normalizer = null,
                IReadOnlyDictionary<string, int>? specialTokens = null,
                bool addPrefixSpace = false,
                bool addBeginningOfSentence = false,
                bool addEndOfSentence = false,
                string? unknownToken = DefaultSpecialToken,
                string? beginningOfSentenceToken = DefaultSpecialToken,
                string? endOfSentenceToken = DefaultSpecialToken) :
            this(vocabularyStream, mergeStream, preTokenizer, normalizer, specialTokens, addPrefixSpace, addBeginningOfSentence, addEndOfSentence, unknownToken, beginningOfSentenceToken, endOfSentenceToken, disposeStream: false)
        {
        }

        private CodeGenTokenizer(Stream vocabularyStream, Stream mergeStream, PreTokenizer? preTokenizer, Normalizer? normalizer, IReadOnlyDictionary<string, int>? specialTokens, bool addPrefixSpace,
                        bool addBeginningOfSentence, bool addEndOfSentence, string? unknownToken, string? beginningOfSentenceToken, string? endOfSentenceToken, bool disposeStream)
        {
            if (vocabularyStream is null)
            {
                throw new ArgumentNullException(nameof(vocabularyStream));
            }

            if (mergeStream is null)
            {
                throw new ArgumentNullException(nameof(mergeStream));
            }

            _preTokenizer = preTokenizer;
            _normalizer = normalizer;

            // Tokenizer data files can be found in codegen-350M-mono
            // https://huggingface.co/Salesforce/codegen-350M-mono/resolve/main/vocab.json?download=true
            // https://huggingface.co/Salesforce/codegen-350M-mono/resolve/main/merges.txt?download=true

            // Or in Phi-2
            // https://huggingface.co/microsoft/phi-2/resolve/main/vocab.json?download=true
            // https://huggingface.co/microsoft/phi-2/resolve/main/merges.txt?download=true

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Pass a non-null, opened Stream for the vocabulary (e.g. File.OpenRead(vocabPath)).
2. If loading an embedded resource, verify the resource name with assembly.GetManifestResourceNames() before calling GetManifestResourceStream.
3. Check the stream-opening call for a null return instead of assuming it always succeeds.

Example fix

// before
var vocab = assembly.GetManifestResourceStream("Wrong.Name.vocab.json"); // null
var tokenizer = new CodeGenTokenizer(vocab, mergesStream);
// after
var vocab = assembly.GetManifestResourceStream("MyApp.Models.vocab.json") ?? throw new InvalidOperationException("Embedded vocab resource not found");
var tokenizer = new CodeGenTokenizer(vocab, mergesStream);

var vocabStream = assembly.GetManifestResourceStream(resourceName);
if (vocabStream is null) throw new InvalidOperationException($"Embedded resource '{resourceName}' not found");

if (vocabStream is Stream vs) { /* construct with vs */ }

try { var tok = new CodeGenTokenizer(vocabStream, mergeStream); } catch (ArgumentNullException ex) { logger.LogError(ex, "Null stream argument: {Param}", ex.ParamName); }

When it happens

Trigger: new CodeGenTokenizer((Stream)null!, mergesStream) or a factory method returning null for the vocab stream, e.g. Assembly.GetManifestResourceStream returning null because the embedded resource name is wrong.

Common situations: Embedded resource typo (namespace/name mismatch) makes GetManifestResourceStream return null; conditional asset loading returned null silently.

Related errors

• throw new ArgumentNullException(nameof(vocabStream));
• throw new ArgumentNullException(nameof(mergeStream));
• throw new ArgumentNullException(nameof(vocabStream));
• throw new ArgumentNullException(nameof(vocabFilePath));
• throw new ArgumentNullException(nameof(vocabulary));

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/c63ca1fd4030abe7. Report an issue: GitHub.