{"record":{"id":"4caf65aba060e920","repo":"dotnet/machinelearning","slug":"failed-to-read-the-vocabulary-file","errorCode":null,"errorMessage":"Failed to read the vocabulary file.","messagePattern":"Failed to read the vocabulary file\\.","errorType":"exception","errorClass":"ArgumentException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs","lineNumber":1761,"sourceCode":"            { \"\\t\\t\\t\",                             50293 },\n            { \"\\t\\t\",                               50294 },\n        };\n\n        private static Dictionary<StringSpanOrdinalKey, (int, string)> GetVocabulary(Stream vocabularyStream)\n        {\n            Vocabulary? vocab;\n            try\n            {\n                vocab = JsonSerializer.Deserialize(vocabularyStream, ModelSourceGenerationContext.Default.Vocabulary);\n            }\n            catch (Exception e)\n            {\n                throw new ArgumentException($\"Problems met when parsing JSON vocabulary object.{Environment.NewLine}Error message: {e.Message}\");\n            }\n\n            if (vocab is null)\n            {\n                throw new ArgumentException($\"Failed to read the vocabulary file.\");\n            }\n\n            return vocab;\n        }\n\n        internal static Dictionary<StringSpanOrdinalKeyPair, int> GetMergeRanks(Stream mergeStream)\n        {\n            var mergeRanks = new Dictionary<StringSpanOrdinalKeyPair, int>();\n            try\n            {\n                using StreamReader reader = new StreamReader(mergeStream);\n\n                // We ignore the first and last line in the file\n                if (reader.Peek() >= 0)\n                {\n                    string ignored = reader.ReadLine()!;\n                }\n","sourceCodeStart":1743,"sourceCodeEnd":1779,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs#L1743-L1779","documentation":"If JsonSerializer.Deserialize returns null for the vocabulary stream, the loader throws ArgumentException('Failed to read the vocabulary file.'). This occurs when the JSON deserializes to null (e.g. the stream contains the literal 'null') or yields no usable vocabulary object.","triggerScenarios":"Passing a vocabulary stream whose JSON payload deserializes to null — an empty-ish file containing 'null', or content that does not map onto the Vocabulary shape expected by ModelSourceGenerationContext.","commonSituations":"Placeholder files checked into source control ('null' or empty), zero-byte downloads, or wrong file supplied as the vocabulary.","solutions":["Inspect the stream content: it must be a valid JSON token-to-id mapping, not 'null' or empty.","Verify the file is the model's vocab.json and re-download if it is empty or a placeholder.","Check stream position/length before passing (Length > 0, Position == 0).","Add a pre-flight JSON deserialization in your code to fail with a clearer message."],"exampleFix":"// before\nusing var s = File.OpenRead(\"vocab.json\"); // 0 bytes -> deserializes to null\n// after\nif (new FileInfo(\"vocab.json\").Length == 0) throw new InvalidOperationException(\"vocab.json is empty\");\nusing var s = File.OpenRead(\"vocab.json\");","handlingStrategy":"validation","validationCode":"if (new FileInfo(vocabPath).Length == 0) throw new InvalidOperationException(\"vocab.json is empty\");\n// or after load: if (JsonSerializer.Deserialize<Vocabulary>(json) is null) throw ...","typeGuard":"bool IsUsableVocab(Stream s) { try { return JsonSerializer.Deserialize<Dictionary<string,int>>(s) is { Count: > 0 }; } catch { return false; } }","tryCatchPattern":"try { var tok = CodeGenTokenizer.Create(vocabStream, mergesStream); }\ncatch (ArgumentException ex) when (ex.Message == \"Failed to read the vocabulary file.\") { /* re-fetch vocab.json */ }","preventionTips":["Check file size and content after download","Reject placeholder/empty files at configuration time","Verify vocab.json deserializes to a non-empty map before creating the tokenizer"],"tags":["tokenizer","vocabulary","file-read","dotnet"],"backgroundTag":"file-read-failed","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}