dotnet/machinelearning · error · ArgumentException

Problems met when parsing JSON vocabulary object.{Environmen

Error message

Problems met when parsing JSON vocabulary object.{Environment.NewLine}Error message: {e.Message}

What it means

Thrown by EnglishRobertaTokenizer.GetVocabulary when the JSON vocabulary stream fails to deserialize into Dictionary<string, int>. The library catches any deserialization exception and rethrows it as ArgumentException, embedding the original error message.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/EnglishRobertaTokenizer.cs:188

            if (disposeStream)
            {
                vocabularyStream.Dispose();
                mergeStream.Dispose();
                highestOccurrenceMappingStream.Dispose();
            }
        }

        private static Dictionary<StringSpanOrdinalKey, int> GetVocabulary(Stream vocabularyStream)
        {
            Dictionary<StringSpanOrdinalKey, int>? vocab;
            try
            {
                vocab = JsonSerializer.Deserialize(vocabularyStream, ModelSourceGenerationContext.Default.DictionaryStringSpanOrdinalKeyInt32);
            }
            catch (Exception e)
            {
                throw new ArgumentException($"Problems met when parsing JSON vocabulary object.{Environment.NewLine}Error message: {e.Message}");
            }

            if (vocab is null)
            {
                throw new ArgumentException($"Failed to read the vocabulary file.");
            }

            return vocab;
        }

        private static Cache<(string, string), int> GetMergeRanks(Stream mergeStream)
        {
            var mergeRanks = new Cache<(string, string), int>(60_000);
            try
            {
                using StreamReader reader = new StreamReader(mergeStream);

                // We ignore the first and last line in the file

View on GitHub (pinned to 7b76e69cf9)

Solutions

Validate the stream contains JSON of the form {"token": id, ...} with integer values before passing it in.
Re-download the correct vocab.json from the model repository and check the file size/hash.
Decode as UTF-8 and run System.Text.Json deserialization locally to see the precise parse error.
Ensure the stream position is at 0 and the stream is fully readable before construction.

Example fix

// before
var tokenizer = new EnglishRobertaTokenizer(vocabStream, mergesStream);
// after
vocabStream.Seek(0, SeekOrigin.Begin);
var check = JsonSerializer.Deserialize<Dictionary<string, int>>(vocabStream); // throws a precise JsonException if malformed
vocabStream.Seek(0, SeekOrigin.Begin);
var tokenizer = new EnglishRobertaTokenizer(vocabStream, mergesStream);

Defensive patterns

Strategy: validation

Validate before calling

vocabStream.Seek(0, SeekOrigin.Begin);
using var doc = JsonDocument.Parse(vocabStream);
if (doc.RootElement.ValueKind != JsonValueKind.Object) throw new InvalidDataException("Vocab must be a JSON object");
foreach (var p in doc.RootElement.EnumerateObject())
    if (p.Value.ValueKind != JsonValueKind.Number) throw new InvalidDataException($"Token '{p.Name}' value is not a number");
vocabStream.Seek(0, SeekOrigin.Begin);

Try / catch

try { var t = new EnglishRobertaTokenizer(vocabStream, mergesStream); }
catch (ArgumentException ex) when (ex.Message.StartsWith("Problems met when parsing")) { throw new InvalidDataException("vocab.json is malformed", ex); }

Prevention

Validate vocab.json is a JSON object of string->int before use.
Pin the vocab file version/hash to the model you load.
Check for HTML error pages saved as .json from failed downloads.
Reset stream position before every use.

When it happens

Trigger: Passing a vocabularyStream whose content is not valid JSON, or JSON whose shape does not match a string->int dictionary (e.g., a JSON array, nested objects, or non-numeric token values).

Common situations: Downloading the wrong vocab.json (e.g., a GPT-2 vocab for a RoBERTa model with different structure); an HTML error page saved as vocab.json; a truncated download; using a case where values are floats or strings.

Understand the failure class

Background: JSON parse error: "Unexpected token" / "not valid JSON" / "failed to parse" — what JSON parsers are really complaining about — this error's family across 45 libraries.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/f0249a1b295499e7. Report an issue: GitHub.