{"record":{"id":"e8b4e92ee0cbd281","repo":"dotnet/machinelearning","slug":"the-vocabulary-cannot-be-empty","errorCode":null,"errorMessage":"The vocabulary cannot be empty.","messagePattern":"The vocabulary cannot be empty\\.","errorType":"exception","errorClass":"InvalidOperationException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs","lineNumber":161,"sourceCode":"            {\n                throw new ArgumentNullException(nameof(options));\n            }\n\n            if (options.Vocabulary is null)\n            {\n                throw new ArgumentNullException(nameof(options.Vocabulary), \"The vocabulary cannot be null.\");\n            }\n\n            Dictionary<StringSpanOrdinalKey, int> vocab = new Dictionary<StringSpanOrdinalKey, int>(1000);\n\n            foreach (KeyValuePair<string, int> kvp in options.Vocabulary)\n            {\n                vocab.Add(new StringSpanOrdinalKey(kvp.Key), kvp.Value);\n            }\n\n            if (vocab.Count == 0)\n            {\n                throw new InvalidOperationException(\"The vocabulary cannot be empty.\");\n            }\n\n            Vec<(string, string)> merges = default;\n            if (options.Merges is not null)\n            {\n                merges = new Vec<(string, string)>(1000);\n\n                foreach (string merge in options.Merges)\n                {\n                    if (merge is null)\n                    {\n                        throw new InvalidOperationException(\"The merge entries cannot be null.\");\n                    }\n\n                    int index = merge.IndexOf(' ');\n                    if (index < 0 || index == merge.Length - 1 || merge.IndexOf(' ', index + 1) >= 0)\n                    {\n                        throw new InvalidOperationException($\"Invalid merger file format\");","sourceCodeStart":143,"sourceCodeEnd":179,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs#L143-L179","documentation":"After copying the provided vocabulary into an internal dictionary, BpeTokenizer.Create(BpeOptions) checks vocab.Count == 0 and throws InvalidOperationException('The vocabulary cannot be empty.'). An empty vocab means every token would be unknown, so construction is rejected.","triggerScenarios":"Passing BpeOptions with an empty (or all-null-key) Vocabulary dictionary, or a Vocabulary deserialized from an empty/misformatted JSON so that zero entries are actually added.","commonSituations":"Pointing the deserializer at an empty or wrong-format vocab file; loading vocab from a stream that was already consumed (position at end); case/key mismatch causing entries to be skipped; building vocab programmatically but the population loop never runs.","solutions":["Verify the vocabulary dictionary has entries before calling Create (vocab.Count > 0).","Fix the vocab file/loading code — check the file is non-empty, correctly formatted, and the deserializer maps keys correctly.","Reset the stream position to 0 if you deserialized from a previously-read stream.","Log/inspect vocab.Count at load time to catch the failure at the source."],"exampleFix":"// before\nvar vocab = JsonSerializer.Deserialize<Dictionary<string,int>>(emptyStream);\nvar tokenizer = BpeTokenizer.Create(new BpeOptions { Vocabulary = vocab }); // throws\n// after\nstream.Position = 0;\nvar vocab = JsonSerializer.Deserialize<Dictionary<string,int>>(stream) ?? throw new InvalidOperationException(\"vocab file empty/invalid\");\nif (vocab.Count == 0) throw new InvalidOperationException(\"vocab file had no tokens\");","handlingStrategy":"validation","validationCode":"if (vocab == null || vocab.Count == 0)\n    throw new InvalidOperationException(\"Vocabulary is empty; check vocab file/loading.\");","typeGuard":"static bool IsNonEmptyVocab(Dictionary<string,int>? v) => v is { Count: > 0 };","tryCatchPattern":"try { var t = BpeTokenizer.Create(options); }\ncatch (InvalidOperationException ex) when (ex.Message.Contains(\"vocabulary cannot be empty\"))\n{ /* reload/repair vocab file, then retry */ }","preventionTips":["Log vocab.Count immediately after loading and assert it is > 0.","Reset stream Position before deserializing from a re-read stream.","Validate vocab file size/format at deployment time."],"tags":["dotnet","tokenizers","bpe","empty-collection"],"backgroundTag":"empty-required-field","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}