{"record":{"id":"cbd1dc150102b129","repo":"dotnet/machinelearning","slug":"failed-to-read-the-vocabulary-file-cbd1dc","errorCode":null,"errorMessage":"Failed to read the vocabulary file.","messagePattern":"Failed to read the vocabulary file\\.","errorType":"exception","errorClass":"ArgumentException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/EnglishRobertaTokenizer.cs","lineNumber":193,"sourceCode":"                highestOccurrenceMappingStream.Dispose();\n            }\n        }\n\n        private static Dictionary<StringSpanOrdinalKey, int> GetVocabulary(Stream vocabularyStream)\n        {\n            Dictionary<StringSpanOrdinalKey, int>? vocab;\n            try\n            {\n                vocab = JsonSerializer.Deserialize(vocabularyStream, ModelSourceGenerationContext.Default.DictionaryStringSpanOrdinalKeyInt32);\n            }\n            catch (Exception e)\n            {\n                throw new ArgumentException($\"Problems met when parsing JSON vocabulary object.{Environment.NewLine}Error message: {e.Message}\");\n            }\n\n            if (vocab is null)\n            {\n                throw new ArgumentException($\"Failed to read the vocabulary file.\");\n            }\n\n            return vocab;\n        }\n\n        private static Cache<(string, string), int> GetMergeRanks(Stream mergeStream)\n        {\n            var mergeRanks = new Cache<(string, string), int>(60_000);\n            try\n            {\n                using StreamReader reader = new StreamReader(mergeStream);\n\n                // We ignore the first and last line in the file\n                if (reader.Peek() >= 0)\n                {\n                    string ignored = reader.ReadLine()!;\n                }\n","sourceCodeStart":175,"sourceCodeEnd":211,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/EnglishRobertaTokenizer.cs#L175-L211","documentation":"Thrown when the vocabulary JSON deserializes successfully but yields a null result (JsonSerializer.Deserialize returns null, e.g. for a stream containing 'null' or an empty payload consumed as JSON null). The library requires a non-null vocabulary dictionary.","triggerScenarios":"Passing an empty vocabularyStream or a stream whose entire content is the JSON literal 'null' to the EnglishRobertaTokenizer constructor.","commonSituations":"A zero-byte vocab.json produced by a failed download; a serialization pipeline that wrote 'null'; passing MemoryStream that was never written to.","solutions":["Check the vocabulary stream length is non-zero and contains a JSON object before constructing the tokenizer.","Re-download or regenerate vocab.json and verify it starts with '{'.","Wrap the stream in a StreamReader and assert the first non-whitespace character is '{'."],"exampleFix":"// before\nvar tokenizer = new EnglishRobertaTokenizer(maybeEmptyStream, mergesStream);\n// after\nif (maybeEmptyStream.Length == 0) throw new InvalidOperationException(\"vocabulary file is empty\");\nvar tokenizer = new EnglishRobertaTokenizer(maybeEmptyStream, mergesStream);","handlingStrategy":"validation","validationCode":"if (vocabStream.Length == 0) throw new InvalidDataException(\"Vocabulary file is empty\");\nvocabStream.Seek(0, SeekOrigin.Begin);\nint first = vocabStream.ReadByte();\nvocabStream.Seek(0, SeekOrigin.Begin);\nif (first != '{') throw new InvalidDataException(\"Vocabulary file is not a JSON object\");","typeGuard":null,"tryCatchPattern":"try { var t = new EnglishRobertaTokenizer(vocabStream, mergesStream); }\ncatch (ArgumentException ex) when (ex.Message.Contains(\"Failed to read the vocabulary file\")) { throw new InvalidDataException(\"vocab.json deserialized to null; check file contents\", ex); }","preventionTips":["Verify vocab file size > 0 at startup.","Reject files whose JSON root is 'null'.","Fail fast on download completion by parsing once before caching.","Log file length and first bytes when tokenizer init fails."],"tags":["json","empty-input","tokenizer","csharp"],"backgroundTag":"empty-required-field","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}