{"record":{"id":"cd323a5d00c603b5","repo":"dotnet/machinelearning","slug":"problems-met-when-parsing-json-vocabulary-object","errorCode":null,"errorMessage":"Problems met when parsing JSON vocabulary object.{Environment.NewLine}Error message: {e.Message}","messagePattern":"Problems met when parsing JSON vocabulary object\\.(.+?)Error message: (.+?)","errorType":"exception","errorClass":"ArgumentException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs","lineNumber":1756,"sourceCode":"            { \"\\t\\t\\t\\t\\t\\t\\t\\t\",                   50288 },\n            { \"\\t\\t\\t\\t\\t\\t\\t\",                     50289 },\n            { \"\\t\\t\\t\\t\\t\\t\",                       50290 },\n            { \"\\t\\t\\t\\t\\t\",                         50291 },\n            { \"\\t\\t\\t\\t\",                           50292 },\n            { \"\\t\\t\\t\",                             50293 },\n            { \"\\t\\t\",                               50294 },\n        };\n\n        private static Dictionary<StringSpanOrdinalKey, (int, string)> GetVocabulary(Stream vocabularyStream)\n        {\n            Vocabulary? vocab;\n            try\n            {\n                vocab = JsonSerializer.Deserialize(vocabularyStream, ModelSourceGenerationContext.Default.Vocabulary);\n            }\n            catch (Exception e)\n            {\n                throw new ArgumentException($\"Problems met when parsing JSON vocabulary object.{Environment.NewLine}Error message: {e.Message}\");\n            }\n\n            if (vocab is null)\n            {\n                throw new ArgumentException($\"Failed to read the vocabulary file.\");\n            }\n\n            return vocab;\n        }\n\n        internal static Dictionary<StringSpanOrdinalKeyPair, int> GetMergeRanks(Stream mergeStream)\n        {\n            var mergeRanks = new Dictionary<StringSpanOrdinalKeyPair, int>();\n            try\n            {\n                using StreamReader reader = new StreamReader(mergeStream);\n\n                // We ignore the first and last line in the file","sourceCodeStart":1738,"sourceCodeEnd":1774,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs#L1738-L1774","documentation":"When loading the vocabulary, CodeGenTokenizer deserializes the stream as JSON via JsonSerializer.Deserialize; any exception during parsing is wrapped and rethrown as ArgumentException with the inner message appended. It means the vocabulary stream content is not the expected JSON object shape.","triggerScenarios":"Calling CodeGenTokenizer.Create with a vocab stream whose body fails JSON deserialization into the Vocabulary type — malformed JSON, wrong structure, non-UTF8 data, or a stream that actually contains the merges.txt content instead of vocab.json.","commonSituations":"Swapping the vocab and merges file paths, downloading an HTML error page instead of the vocab file, truncated downloads, or BOM/encoding issues.","solutions":["Verify you are passing vocab.json (the token-to-id JSON object), not merges.txt, as the vocabulary stream.","Open the stream content and validate the JSON (e.g. deserialize it yourself first) to see the underlying parse error.","Re-download the vocabulary file and confirm it is complete and UTF-8 encoded.","Check that the stream is positioned at 0 (seek to beginning) before passing it in."],"exampleFix":"// before\nvar tok = CodeGenTokenizer.Create(mergesStream, vocabStream); // swapped\n// after\nvar tok = CodeGenTokenizer.Create(vocabStream, mergesStream);","handlingStrategy":"try-catch","validationCode":"// Pre-flight: parse the vocab yourself\nusing var doc = JsonDocument.Parse(vocabJson);\nif (doc.RootElement.ValueKind != JsonValueKind.Object || doc.RootElement.EnumerateObject().Any()) { /* shape ok */ }","typeGuard":"bool LooksLikeVocabJson(Stream s) { try { using var d = JsonDocument.Parse(s); return d.RootElement.ValueKind == JsonValueKind.Object; } catch { return false; } }","tryCatchPattern":"try { var tok = CodeGenTokenizer.Create(vocabStream, mergesStream); }\ncatch (ArgumentException ex) when (ex.Message.StartsWith(\"Problems met when parsing JSON vocabulary\")) { /* reload/repair vocab.json */ }","preventionTips":["Confirm which file is vocab.json vs merges.txt before wiring streams","Validate downloaded model files (size, JSON parse) before use","Seek stream position to 0 before passing"],"tags":["tokenizer","json","vocabulary","deserialization","dotnet"],"backgroundTag":"json-parse-error","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}