dotnet/machinelearning · error · ArgumentException

Problems met when parsing JSON vocabulary object.{Environmen

Error message

Problems met when parsing JSON vocabulary object.{Environment.NewLine}Error message: {e.Message}

What it means

When loading the vocabulary, CodeGenTokenizer deserializes the stream as JSON via JsonSerializer.Deserialize; any exception during parsing is wrapped and rethrown as ArgumentException with the inner message appended. It means the vocabulary stream content is not the expected JSON object shape.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs:1756

            { "\t\t\t\t\t\t\t\t",                   50288 },
            { "\t\t\t\t\t\t\t",                     50289 },
            { "\t\t\t\t\t\t",                       50290 },
            { "\t\t\t\t\t",                         50291 },
            { "\t\t\t\t",                           50292 },
            { "\t\t\t",                             50293 },
            { "\t\t",                               50294 },
        };

        private static Dictionary<StringSpanOrdinalKey, (int, string)> GetVocabulary(Stream vocabularyStream)
        {
            Vocabulary? vocab;
            try
            {
                vocab = JsonSerializer.Deserialize(vocabularyStream, ModelSourceGenerationContext.Default.Vocabulary);
            }
            catch (Exception e)
            {
                throw new ArgumentException($"Problems met when parsing JSON vocabulary object.{Environment.NewLine}Error message: {e.Message}");
            }

            if (vocab is null)
            {
                throw new ArgumentException($"Failed to read the vocabulary file.");
            }

            return vocab;
        }

        internal static Dictionary<StringSpanOrdinalKeyPair, int> GetMergeRanks(Stream mergeStream)
        {
            var mergeRanks = new Dictionary<StringSpanOrdinalKeyPair, int>();
            try
            {
                using StreamReader reader = new StreamReader(mergeStream);

                // We ignore the first and last line in the file

View on GitHub (pinned to 7b76e69cf9)

Solutions

Verify you are passing vocab.json (the token-to-id JSON object), not merges.txt, as the vocabulary stream.
Open the stream content and validate the JSON (e.g. deserialize it yourself first) to see the underlying parse error.
Re-download the vocabulary file and confirm it is complete and UTF-8 encoded.
Check that the stream is positioned at 0 (seek to beginning) before passing it in.

Example fix

// before
var tok = CodeGenTokenizer.Create(mergesStream, vocabStream); // swapped
// after
var tok = CodeGenTokenizer.Create(vocabStream, mergesStream);

Defensive patterns

Strategy: try-catch

Validate before calling

// Pre-flight: parse the vocab yourself
using var doc = JsonDocument.Parse(vocabJson);
if (doc.RootElement.ValueKind != JsonValueKind.Object || doc.RootElement.EnumerateObject().Any()) { /* shape ok */ }

Type guard

bool LooksLikeVocabJson(Stream s) { try { using var d = JsonDocument.Parse(s); return d.RootElement.ValueKind == JsonValueKind.Object; } catch { return false; } }

Try / catch

try { var tok = CodeGenTokenizer.Create(vocabStream, mergesStream); }
catch (ArgumentException ex) when (ex.Message.StartsWith("Problems met when parsing JSON vocabulary")) { /* reload/repair vocab.json */ }

Prevention

Confirm which file is vocab.json vs merges.txt before wiring streams
Validate downloaded model files (size, JSON parse) before use
Seek stream position to 0 before passing

When it happens

Trigger: Calling CodeGenTokenizer.Create with a vocab stream whose body fails JSON deserialization into the Vocabulary type — malformed JSON, wrong structure, non-UTF8 data, or a stream that actually contains the merges.txt content instead of vocab.json.

Common situations: Swapping the vocab and merges file paths, downloading an HTML error page instead of the vocab file, truncated downloads, or BOM/encoding issues.

Understand the failure class

Background: JSON parse error: "Unexpected token" / "not valid JSON" / "failed to parse" — what JSON parsers are really complaining about — this error's family across 45 libraries.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/cd323a5d00c603b5. Report an issue: GitHub.