dotnet/machinelearning · error · InvalidOperationException

The vocabulary cannot be empty.

Error message

The vocabulary cannot be empty.

What it means

After copying the provided vocabulary into an internal dictionary, BpeTokenizer.Create(BpeOptions) checks vocab.Count == 0 and throws InvalidOperationException('The vocabulary cannot be empty.'). An empty vocab means every token would be unknown, so construction is rejected.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs:161

            {
                throw new ArgumentNullException(nameof(options));
            }

            if (options.Vocabulary is null)
            {
                throw new ArgumentNullException(nameof(options.Vocabulary), "The vocabulary cannot be null.");
            }

            Dictionary<StringSpanOrdinalKey, int> vocab = new Dictionary<StringSpanOrdinalKey, int>(1000);

            foreach (KeyValuePair<string, int> kvp in options.Vocabulary)
            {
                vocab.Add(new StringSpanOrdinalKey(kvp.Key), kvp.Value);
            }

            if (vocab.Count == 0)
            {
                throw new InvalidOperationException("The vocabulary cannot be empty.");
            }

            Vec<(string, string)> merges = default;
            if (options.Merges is not null)
            {
                merges = new Vec<(string, string)>(1000);

                foreach (string merge in options.Merges)
                {
                    if (merge is null)
                    {
                        throw new InvalidOperationException("The merge entries cannot be null.");
                    }

                    int index = merge.IndexOf(' ');
                    if (index < 0 || index == merge.Length - 1 || merge.IndexOf(' ', index + 1) >= 0)
                    {
                        throw new InvalidOperationException($"Invalid merger file format");

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Verify the vocabulary dictionary has entries before calling Create (vocab.Count > 0).
2. Fix the vocab file/loading code — check the file is non-empty, correctly formatted, and the deserializer maps keys correctly.
3. Reset the stream position to 0 if you deserialized from a previously-read stream.
4. Log/inspect vocab.Count at load time to catch the failure at the source.

Example fix

// before
var vocab = JsonSerializer.Deserialize<Dictionary<string,int>>(emptyStream);
var tokenizer = BpeTokenizer.Create(new BpeOptions { Vocabulary = vocab }); // throws
// after
stream.Position = 0;
var vocab = JsonSerializer.Deserialize<Dictionary<string,int>>(stream) ?? throw new InvalidOperationException("vocab file empty/invalid");
if (vocab.Count == 0) throw new InvalidOperationException("vocab file had no tokens");

if (vocab == null || vocab.Count == 0)
    throw new InvalidOperationException("Vocabulary is empty; check vocab file/loading.");

static bool IsNonEmptyVocab(Dictionary<string,int>? v) => v is { Count: > 0 };

try { var t = BpeTokenizer.Create(options); }
catch (InvalidOperationException ex) when (ex.Message.Contains("vocabulary cannot be empty"))
{ /* reload/repair vocab file, then retry */ }

When it happens

Trigger: Passing BpeOptions with an empty (or all-null-key) Vocabulary dictionary, or a Vocabulary deserialized from an empty/misformatted JSON so that zero entries are actually added.

Common situations: Pointing the deserializer at an empty or wrong-format vocab file; loading vocab from a stream that was already consumed (position at end); case/key mismatch causing entries to be skipped; building vocab programmatically but the population loop never runs.

Understand the failure class

Background: "must not be empty", "cannot be empty" — required-field validation errors across open-source libraries — this error's family across 41 libraries.

Related errors

• Unknown Token '{value}' was not present in '{nameof(Vocabula
• The vocabulary cannot be null.
• The merge entries cannot be null.
• Invalid merger file format
• Must provide at least 1 ID column

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/e8b4e92ee0cbd281. Report an issue: GitHub.