dotnet/machinelearning · error · ArgumentNullException

The vocabulary cannot be null.

Error message

The vocabulary cannot be null.

What it means

BpeTokenizer.Create(BpeOptions) requires options.Vocabulary to be a non-null dictionary mapping tokens to ids; a null Vocabulary throws ArgumentNullException with message 'The vocabulary cannot be null.' The BPE model cannot encode/decode without token-id assignments.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs:149

            return new BpeTokenizer(result.vocab, result.merges, preTokenizer, normalizer, specialTokens, unknownToken, continuingSubwordPrefix, endOfWordSuffix, fuseUnknownTokens);
        }

        /// <summary>
        /// Create a new Bpe tokenizer object to use for text encoding.
        /// </summary>
        /// <param name="options">The options used to create the Bpe tokenizer.</param>
        /// <returns>The Bpe tokenizer object.</returns>
        public static BpeTokenizer Create(BpeOptions options)
        {
            if (options is null)
            {
                throw new ArgumentNullException(nameof(options));
            }

            if (options.Vocabulary is null)
            {
                throw new ArgumentNullException(nameof(options.Vocabulary), "The vocabulary cannot be null.");
            }

            Dictionary<StringSpanOrdinalKey, int> vocab = new Dictionary<StringSpanOrdinalKey, int>(1000);

            foreach (KeyValuePair<string, int> kvp in options.Vocabulary)
            {
                vocab.Add(new StringSpanOrdinalKey(kvp.Key), kvp.Value);
            }

            if (vocab.Count == 0)
            {
                throw new InvalidOperationException("The vocabulary cannot be empty.");
            }

            Vec<(string, string)> merges = default;
            if (options.Merges is not null)
            {
                merges = new Vec<(string, string)>(1000);

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Populate options.Vocabulary with a Dictionary<string,int> of token->id before calling Create.
2. If loading from a file, use the file-based Create(vocabFile, mergesFile) overload instead.
3. Check that deserialization/config binding actually fills Vocabulary (key names, case sensitivity).
4. Guard: if (options?.Vocabulary == null) load or throw with a clear message before Create.

Example fix

// before
var options = new BpeOptions { Merges = mergesList }; // Vocabulary null
var tokenizer = BpeTokenizer.Create(options);
// after
var options = new BpeOptions { Merges = mergesList, Vocabulary = vocabDict };
var tokenizer = BpeTokenizer.Create(options);

if (options?.Vocabulary == null || options.Vocabulary.Count == 0)
    throw new InvalidOperationException("BpeOptions.Vocabulary must be non-null and non-empty.");

static bool HasVocabulary(BpeOptions? o) => o?.Vocabulary is { Count: > 0 };

try { var t = BpeTokenizer.Create(options); }
catch (ArgumentNullException ex) when (ex.Message.Contains("vocabulary cannot be null"))
{ options.Vocabulary = LoadVocabFromDisk(); var t = BpeTokenizer.Create(options); }

When it happens

Trigger: Passing a BpeOptions whose Vocabulary property was never assigned (default null) to BpeTokenizer.Create — e.g. setting only Merges or file paths on the options object.

Common situations: Assuming Vocabulary can be loaded lazily or from VocabularyFile via the options object (it cannot in this overload); partial initialization after refactoring; deserialization leaving Vocabulary null.

Related errors

• Unknown Token '{value}' was not present in '{nameof(Vocabula
• The vocabulary cannot be empty.
• The merge entries cannot be null.
• Invalid merger file format
• The beginning of sentence token '{beginningOfSentenceToken}'

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/ce255b049e557916. Report an issue: GitHub.