dotnet/machinelearning · error · ArgumentNullException

throw new ArgumentNullException(nameof(mergeStream));

Error message

throw new ArgumentNullException(nameof(mergeStream));

What it means

The private CodeGenTokenizer constructor throws ArgumentNullException when mergeStream is null. Both the vocabulary stream and the merges stream must be supplied since the constructor reads the BPE merge rules from them. The check follows the vocabularyStream check, so both arguments must be non-null streams.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs:113

                bool addEndOfSentence = false,
                string? unknownToken = DefaultSpecialToken,
                string? beginningOfSentenceToken = DefaultSpecialToken,
                string? endOfSentenceToken = DefaultSpecialToken) :
            this(vocabularyStream, mergeStream, preTokenizer, normalizer, specialTokens, addPrefixSpace, addBeginningOfSentence, addEndOfSentence, unknownToken, beginningOfSentenceToken, endOfSentenceToken, disposeStream: false)
        {
        }

        private CodeGenTokenizer(Stream vocabularyStream, Stream mergeStream, PreTokenizer? preTokenizer, Normalizer? normalizer, IReadOnlyDictionary<string, int>? specialTokens, bool addPrefixSpace,
                        bool addBeginningOfSentence, bool addEndOfSentence, string? unknownToken, string? beginningOfSentenceToken, string? endOfSentenceToken, bool disposeStream)
        {
            if (vocabularyStream is null)
            {
                throw new ArgumentNullException(nameof(vocabularyStream));
            }

            if (mergeStream is null)
            {
                throw new ArgumentNullException(nameof(mergeStream));
            }

            _preTokenizer = preTokenizer;
            _normalizer = normalizer;

            // Tokenizer data files can be found in codegen-350M-mono
            // https://huggingface.co/Salesforce/codegen-350M-mono/resolve/main/vocab.json?download=true
            // https://huggingface.co/Salesforce/codegen-350M-mono/resolve/main/merges.txt?download=true

            // Or in Phi-2
            // https://huggingface.co/microsoft/phi-2/resolve/main/vocab.json?download=true
            // https://huggingface.co/microsoft/phi-2/resolve/main/merges.txt?download=true

            _vocab = GetVocabulary(vocabularyStream);
            _vocabReverse = _vocab.ToDictionary(kvp => kvp.Value.Id, kvp => kvp.Value.Token);
            _mergeRanks = GetMergeRanks(mergeStream);
            _cache = new StringSpanOrdinalKeyCache<List<EncodedToken>>();

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Pass a non-null opened Stream for merges.txt as the second stream argument.
2. Ensure both streams are opened together, e.g. File.OpenRead for vocab and merges in the same factory method.
3. Verify embedded resource names for the merges asset if loading from an assembly.

Example fix

// before
var tokenizer = new CodeGenTokenizer(File.OpenRead(vocabPath), mergesStream); // mergesStream was never assigned
// after
using var vocabStream = File.OpenRead(vocabPath);
using var mergesStream = File.OpenRead(mergesPath);
var tokenizer = new CodeGenTokenizer(vocabStream, mergesStream);

using var vocabStream = File.OpenRead(vocabPath);
using var mergeStream = File.OpenRead(mergesPath);
if (vocabStream is null || mergeStream is null) throw new InvalidOperationException("Tokenizer streams must be opened");

if (mergeStream is Stream ms) { /* construct with ms */ }

try { var tok = new CodeGenTokenizer(vocabStream, mergeStream); } catch (ArgumentNullException ex) { logger.LogError(ex, "Null stream argument: {Param}", ex.ParamName); }

When it happens

Trigger: new CodeGenTokenizer(vocabStream, null) via the public stream constructor; a helper that opens the vocab file succeeded but the merges File.OpenRead was skipped or its result not assigned.

Common situations: Resource-loading code that opens two streams returns null for the second due to a missing embedded resource; refactoring dropped the merges stream initialization.

Related errors

• throw new ArgumentNullException(nameof(vocabStream));
• throw new ArgumentNullException(nameof(vocabularyStream));
• throw new ArgumentNullException(nameof(vocabStream));
• throw new ArgumentNullException(nameof(vocabFilePath));
• throw new ArgumentNullException(nameof(vocabulary));

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/866be973bfc6e049. Report an issue: GitHub.