dotnet/machinelearning · error · InvalidOperationException

The vocabulary cannot be empty.

Error message

The vocabulary cannot be empty.

What it means

After copying the provided vocabulary into an internal dictionary, BpeTokenizer.Create(BpeOptions) checks vocab.Count == 0 and throws InvalidOperationException('The vocabulary cannot be empty.'). An empty vocab means every token would be unknown, so construction is rejected.

Solutions

1. Verify the vocabulary dictionary has entries before calling Create (vocab.Count > 0).
2. Fix the vocab file/loading code — check the file is non-empty, correctly formatted, and the deserializer maps keys correctly.
3. Reset the stream position to 0 if you deserialized from a previously-read stream.
4. Log/inspect vocab.Count at load time to catch the failure at the source.

Example fix

// before
var vocab = JsonSerializer.Deserialize<Dictionary<string,int>>(emptyStream);
var tokenizer = BpeTokenizer.Create(new BpeOptions { Vocabulary = vocab }); // throws
// after
stream.Position = 0;
var vocab = JsonSerializer.Deserialize<Dictionary<string,int>>(stream) ?? throw new InvalidOperationException("vocab file empty/invalid");
if (vocab.Count == 0) throw new InvalidOperationException("vocab file had no tokens");

if (vocab == null || vocab.Count == 0)
    throw new InvalidOperationException("Vocabulary is empty; check vocab file/loading.");

static bool IsNonEmptyVocab(Dictionary<string,int>? v) => v is { Count: > 0 };

try { var t = BpeTokenizer.Create(options); }
catch (InvalidOperationException ex) when (ex.Message.Contains("vocabulary cannot be empty"))
{ /* reload/repair vocab file, then retry */ }

When it happens

Trigger: Passing BpeOptions with an empty (or all-null-key) Vocabulary dictionary, or a Vocabulary deserialized from an empty/misformatted JSON so that zero entries are actually added.

Common situations: Pointing the deserializer at an empty or wrong-format vocab file; loading vocab from a stream that was already consumed (position at end); case/key mismatch causing entries to be skipped; building vocab programmatically but the population loop never runs.

Understand the failure class

Background: "must not be empty", "cannot be empty" — required-field validation errors across open-source libraries — this error's family across 41 libraries.

Related errors

• Invalid merger file format
• The merge entries cannot be null.
• The vocabulary cannot be null.
• Unknown Token ' ' was not present in ' '.
• Invalid format of merge file at line

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/e8b4e92ee0cbd281. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs:161

            {
                throw new ArgumentNullException(nameof(options));
            }

            if (options.Vocabulary is null)
            {
                throw new ArgumentNullException(nameof(options.Vocabulary), "The vocabulary cannot be null.");
            }

            Dictionary<StringSpanOrdinalKey, int> vocab = new Dictionary<StringSpanOrdinalKey, int>(1000);

            foreach (KeyValuePair<string, int> kvp in options.Vocabulary)
            {
                vocab.Add(new StringSpanOrdinalKey(kvp.Key), kvp.Value);
            }

            if (vocab.Count == 0)
            {
                throw new InvalidOperationException("The vocabulary cannot be empty.");
            }

            Vec<(string, string)> merges = default;
            if (options.Merges is not null)
            {
                merges = new Vec<(string, string)>(1000);

                foreach (string merge in options.Merges)
                {
                    if (merge is null)
                    {
                        throw new InvalidOperationException("The merge entries cannot be null.");
                    }

                    int index = merge.IndexOf(' ');
                    if (index < 0 || index == merge.Length - 1 || merge.IndexOf(' ', index + 1) >= 0)
                    {
                        throw new InvalidOperationException($"Invalid merger file format");

View on GitHub (pinned to 7b76e69cf9)