{"record":{"id":"f0249a1b295499e7","repo":"dotnet/machinelearning","slug":"problems-met-when-parsing-json-vocabulary-object-f0249a","errorCode":null,"errorMessage":"Problems met when parsing JSON vocabulary object.{Environment.NewLine}Error message: {e.Message}","messagePattern":"Problems met when parsing JSON vocabulary object\\.(.+?)Error message: (.+?)","errorType":"exception","errorClass":"ArgumentException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/EnglishRobertaTokenizer.cs","lineNumber":188,"sourceCode":"\n            if (disposeStream)\n            {\n                vocabularyStream.Dispose();\n                mergeStream.Dispose();\n                highestOccurrenceMappingStream.Dispose();\n            }\n        }\n\n        private static Dictionary<StringSpanOrdinalKey, int> GetVocabulary(Stream vocabularyStream)\n        {\n            Dictionary<StringSpanOrdinalKey, int>? vocab;\n            try\n            {\n                vocab = JsonSerializer.Deserialize(vocabularyStream, ModelSourceGenerationContext.Default.DictionaryStringSpanOrdinalKeyInt32);\n            }\n            catch (Exception e)\n            {\n                throw new ArgumentException($\"Problems met when parsing JSON vocabulary object.{Environment.NewLine}Error message: {e.Message}\");\n            }\n\n            if (vocab is null)\n            {\n                throw new ArgumentException($\"Failed to read the vocabulary file.\");\n            }\n\n            return vocab;\n        }\n\n        private static Cache<(string, string), int> GetMergeRanks(Stream mergeStream)\n        {\n            var mergeRanks = new Cache<(string, string), int>(60_000);\n            try\n            {\n                using StreamReader reader = new StreamReader(mergeStream);\n\n                // We ignore the first and last line in the file","sourceCodeStart":170,"sourceCodeEnd":206,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/EnglishRobertaTokenizer.cs#L170-L206","documentation":"Thrown by EnglishRobertaTokenizer.GetVocabulary when the JSON vocabulary stream fails to deserialize into Dictionary<string, int>. The library catches any deserialization exception and rethrows it as ArgumentException, embedding the original error message.","triggerScenarios":"Passing a vocabularyStream whose content is not valid JSON, or JSON whose shape does not match a string->int dictionary (e.g., a JSON array, nested objects, or non-numeric token values).","commonSituations":"Downloading the wrong vocab.json (e.g., a GPT-2 vocab for a RoBERTa model with different structure); an HTML error page saved as vocab.json; a truncated download; using a case where values are floats or strings.","solutions":["Validate the stream contains JSON of the form {\"token\": id, ...} with integer values before passing it in.","Re-download the correct vocab.json from the model repository and check the file size/hash.","Decode as UTF-8 and run System.Text.Json deserialization locally to see the precise parse error.","Ensure the stream position is at 0 and the stream is fully readable before construction."],"exampleFix":"// before\nvar tokenizer = new EnglishRobertaTokenizer(vocabStream, mergesStream);\n// after\nvocabStream.Seek(0, SeekOrigin.Begin);\nvar check = JsonSerializer.Deserialize<Dictionary<string, int>>(vocabStream); // throws a precise JsonException if malformed\nvocabStream.Seek(0, SeekOrigin.Begin);\nvar tokenizer = new EnglishRobertaTokenizer(vocabStream, mergesStream);","handlingStrategy":"validation","validationCode":"vocabStream.Seek(0, SeekOrigin.Begin);\nusing var doc = JsonDocument.Parse(vocabStream);\nif (doc.RootElement.ValueKind != JsonValueKind.Object) throw new InvalidDataException(\"Vocab must be a JSON object\");\nforeach (var p in doc.RootElement.EnumerateObject())\n    if (p.Value.ValueKind != JsonValueKind.Number) throw new InvalidDataException($\"Token '{p.Name}' value is not a number\");\nvocabStream.Seek(0, SeekOrigin.Begin);","typeGuard":null,"tryCatchPattern":"try { var t = new EnglishRobertaTokenizer(vocabStream, mergesStream); }\ncatch (ArgumentException ex) when (ex.Message.StartsWith(\"Problems met when parsing\")) { throw new InvalidDataException(\"vocab.json is malformed\", ex); }","preventionTips":["Validate vocab.json is a JSON object of string->int before use.","Pin the vocab file version/hash to the model you load.","Check for HTML error pages saved as .json from failed downloads.","Reset stream position before every use."],"tags":["json","deserialization","tokenizer","csharp"],"backgroundTag":"json-parse-error","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}