dotnet/machinelearning · error · ArgumentNullException

ArgumentNullException

Error message

ArgumentNullException

What it means

Null-argument guard inside Phi2Tokenizer.Create: while building the CodeGen/Phi-2 tokenizer from the vocab, merges and configuration streams, one of the required arguments (the stream/JSON configuration or vocabulary source) is null, and an ArgumentNullException is thrown. The Create method refuses to proceed because the tokenizer's BPE tables cannot be loaded from nothing.

Solutions

1. Pass a valid non-null Stream for the vocab file (e.g. File.OpenRead("vocab.json"))
2. Check that the embedded resource name is correct when using GetManifestResourceStream
3. Null-check streams before calling Create and surface a clear load error

Example fix

// before
Stream? vocab = assembly.GetManifestResourceStream("Wrong.Name.vocab.json"); // null
var tok = Phi2Tokenizer.Create(vocab, merges);
// after
using Stream vocab = File.OpenRead("vocab.json");
var tok = Phi2Tokenizer.Create(vocab, merges);

if (vocabStream is null) throw new InvalidOperationException("Vocab stream failed to load; check the resource name/path.");

try { var tok = Phi2Tokenizer.Create(vocabStream, mergesStream); } catch (ArgumentNullException ex) { log.LogError(ex, "vocabStream was null"); throw; }

When it happens

Trigger: Calling Phi2Tokenizer.Create(vocabStream: null, mergesStream, ...) or passing a variable that was never assigned because the vocab resource failed to load.

Common situations: Assembly resource name typo so GetManifestResourceStream returns null; file not found handled by returning null stream; refactor renamed a variable.

Related errors

• ArgumentNullException
• A tokenizer.json normalizer entry must be a JSON object.
• Cannot parse the line
• getter
• Incorrect vocabulary format, expected

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/22a66bfd22fbe904. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.ML.Tokenizers/Model/Phi2Tokenizer.cs:108

        /// <returns>The CodeGen tokenizer object.</returns>
        /// <remarks>
        /// The tokenizer will be created according to the configuration specified in https://huggingface.co/microsoft/phi-2/raw/main/tokenizer.json.
        /// It is important to provide the similar vocab and merges files to the ones used in the training of the model.
        /// The vocab and merges files can be downloaded from the following links:
        ///     https://huggingface.co/microsoft/phi-2/resolve/main/vocab.json?download=true
        ///     https://huggingface.co/microsoft/phi-2/resolve/main/merges.txt?download=true
        /// When creating the tokenizer, ensure that the vocabulary stream is sourced from a trusted provider.
        /// </remarks>
        public static new Phi2Tokenizer Create(
            Stream vocabStream,
            Stream mergesStream,
            bool addPrefixSpace = false,
            bool addBeginOfSentence = false,
            bool addEndOfSentence = false)
        {
            if (vocabStream is null)
            {
                throw new ArgumentNullException(nameof(vocabStream));
            }

            if (mergesStream is null)
            {
                throw new ArgumentNullException(nameof(mergesStream));
            }

            return new Phi2Tokenizer(
                        vocabStream, mergesStream, new RegexPreTokenizer(TiktokenTokenizer.P50kBaseRegex(), CodeGenTokenizer.CodeGenSpecialTokens), normalizer: null,
                        CodeGenTokenizer.CodeGenSpecialTokens, addPrefixSpace: addPrefixSpace, addBeginningOfSentence: addBeginOfSentence, addEndOfSentence: addEndOfSentence);
        }
    }
}

View on GitHub (pinned to 7b76e69cf9)