dotnet/machinelearning · critical · ArgumentNullException

throw new ArgumentNullException(nameof(vocabStream));

Error message

throw new ArgumentNullException(nameof(vocabStream));

What it means

BPETokenizer.Create requires a non-null vocabulary stream because the BPE model cannot be constructed without loading vocab data from it. The library throws ArgumentNullException synchronously at the start of the Create method to fail fast on a missing required argument. Passing null means the tokenizer would have no token-to-id mapping at all.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs:239

        /// <param name="endOfWordSuffix">The suffix to attach to sub-word units that represent an end of word.</param>
        /// <param name="fuseUnknownTokens">Indicate whether allowing multiple unknown tokens get fused.</param>
        /// <remarks>
        /// When creating the tokenizer, ensure that the vocabulary stream is sourced from a trusted provider.
        /// </remarks>
        public static BpeTokenizer Create(
                                Stream vocabStream,
                                Stream? mergesStream,
                                PreTokenizer? preTokenizer = null,
                                Normalizer? normalizer = null,
                                IReadOnlyDictionary<string, int>? specialTokens = null,
                                string? unknownToken = null,
                                string? continuingSubwordPrefix = null,
                                string? endOfWordSuffix = null,
                                bool fuseUnknownTokens = false)
        {
            if (vocabStream is null)
            {
                throw new ArgumentNullException(nameof(vocabStream));
            }

            (Dictionary<StringSpanOrdinalKey, int>? vocab, Vec<(string, string)> merges) result = ReadModelDataAsync(vocabStream, mergesStream, useAsync: false).GetAwaiter().GetResult();

            return new BpeTokenizer(result.vocab, result.merges, preTokenizer, normalizer, specialTokens, unknownToken, continuingSubwordPrefix, endOfWordSuffix, fuseUnknownTokens);
        }

        /// <summary>
        /// Create a new Bpe tokenizer object asynchronously to use for text encoding.
        /// </summary>
        /// <param name="vocabStream">The JSON stream containing the dictionary of string keys and their ids.</param>
        /// <param name="mergesStream">The stream containing the tokens's pairs list.</param>
        /// <param name="preTokenizer">The pre-tokenizer to use.</param>
        /// <param name="normalizer">The normalizer to use.</param>
        /// <param name="specialTokens">The dictionary mapping special tokens to Ids.</param>
        /// <param name="unknownToken"> The unknown token to be used by the model.</param>
        /// <param name="continuingSubwordPrefix">The prefix to attach to sub-word units that don’t represent a beginning of word.</param>
        /// <param name="endOfWordSuffix">The suffix to attach to sub-word units that represent an end of word.</param>

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Ensure the vocabulary stream is non-null before calling Create (open the file or embed the resource).
2. Check that the config/env value holding the vocab file path is set and the file exists before opening the stream.
3. If you intend async loading, use CreateAsync with the same non-null guarantee.
4. Wrap stream creation so a missing file throws a clear FileNotFoundException instead of surfacing as a null stream later.

Example fix

// before
using var vocab = File.OpenRead(config.VocabPath); // VocabPath may be null
var tok = BpeTokenizer.Create(vocab, mergesStream);
// after
if (string.IsNullOrEmpty(config.VocabPath)) throw new InvalidOperationException("Vocab path not configured");
using var vocab = File.OpenRead(config.VocabPath);
var tok = BpeTokenizer.Create(vocab ?? throw new InvalidOperationException("vocab stream missing"), mergesStream);

if (vocabStream is null) throw new InvalidOperationException("Vocabulary stream must be provided before calling BpeTokenizer.Create");
var tokenizer = BpeTokenizer.Create(vocabStream, mergesStream);

static bool HasVocabStream(Stream? s) => s is { CanRead: true };

try { var tok = BpeTokenizer.Create(vocabStream, mergesStream); }
catch (ArgumentNullException ex) when (ex.ParamName == "vocabStream") { throw new InvalidOperationException("BPE vocab stream was null; check model file loading", ex); }

When it happens

Trigger: Calling BpeTokenizer.Create(null, mergesStream, ...) or building the vocabStream argument from an expression (e.g. File.OpenRead on a null-configured path wrapped in a helper) that evaluates to null at runtime.

Common situations: Loading the vocab file path from configuration/environment variables that are unset; a helper method returning Stream? that returns null when the file is missing; refactoring where the vocab stream was moved to an optional parameter; deserializing model settings where the vocab file entry is absent.

Related errors

• throw new ArgumentNullException(nameof(vocabulary));
• throw new ArgumentNullException(nameof(vocabFilePath));
• throw new ArgumentNullException(nameof(vocabStream));
• throw new ArgumentNullException(nameof(vocabFile));
• throw new ArgumentNullException(nameof(vocabularyPath));

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/4804a7147c97ac58. Report an issue: GitHub.