dotnet/machinelearning · error · ArgumentException

Problems met when parsing JSON vocabulary object.

Error message

Problems met when parsing JSON vocabulary object.{Environment.NewLine}Error message: {e.Message}

What it means

When loading the vocabulary, CodeGenTokenizer deserializes the stream as JSON via JsonSerializer.Deserialize; any exception during parsing is wrapped and rethrown as ArgumentException with the inner message appended. It means the vocabulary stream content is not the expected JSON object shape.

Solutions

1. Verify you are passing vocab.json (the token-to-id JSON object), not merges.txt, as the vocabulary stream.
2. Open the stream content and validate the JSON (e.g. deserialize it yourself first) to see the underlying parse error.
3. Re-download the vocabulary file and confirm it is complete and UTF-8 encoded.
4. Check that the stream is positioned at 0 (seek to beginning) before passing it in.

Example fix

// before
var tok = CodeGenTokenizer.Create(mergesStream, vocabStream); // swapped
// after
var tok = CodeGenTokenizer.Create(vocabStream, mergesStream);

// Pre-flight: parse the vocab yourself
using var doc = JsonDocument.Parse(vocabJson);
if (doc.RootElement.ValueKind != JsonValueKind.Object || doc.RootElement.EnumerateObject().Any()) { /* shape ok */ }

bool LooksLikeVocabJson(Stream s) { try { using var d = JsonDocument.Parse(s); return d.RootElement.ValueKind == JsonValueKind.Object; } catch { return false; } }

try { var tok = CodeGenTokenizer.Create(vocabStream, mergesStream); }
catch (ArgumentException ex) when (ex.Message.StartsWith("Problems met when parsing JSON vocabulary")) { /* reload/repair vocab.json */ }

When it happens

Trigger: Calling CodeGenTokenizer.Create with a vocab stream whose body fails JSON deserialization into the Vocabulary type — malformed JSON, wrong structure, non-UTF8 data, or a stream that actually contains the merges.txt content instead of vocab.json.

Common situations: Swapping the vocab and merges file paths, downloading an HTML error page instead of the vocab file, truncated downloads, or BOM/encoding issues.

Understand the failure class

Background: JSON parse error: "Unexpected token" / "not valid JSON" / "failed to parse" — what JSON parsers are really complaining about — this error's family across 45 libraries.

Related errors

• Failed to read the vocabulary file.
• Invalid JSON.
• Problems met when parsing JSON vocabulary object.
• The beginning of sentence token
• The end of sentence token

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/cd323a5d00c603b5. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs:1756

            { "\t\t\t\t\t\t\t\t",                   50288 },
            { "\t\t\t\t\t\t\t",                     50289 },
            { "\t\t\t\t\t\t",                       50290 },
            { "\t\t\t\t\t",                         50291 },
            { "\t\t\t\t",                           50292 },
            { "\t\t\t",                             50293 },
            { "\t\t",                               50294 },
        };

        private static Dictionary<StringSpanOrdinalKey, (int, string)> GetVocabulary(Stream vocabularyStream)
        {
            Vocabulary? vocab;
            try
            {
                vocab = JsonSerializer.Deserialize(vocabularyStream, ModelSourceGenerationContext.Default.Vocabulary);
            }
            catch (Exception e)
            {
                throw new ArgumentException($"Problems met when parsing JSON vocabulary object.{Environment.NewLine}Error message: {e.Message}");
            }

            if (vocab is null)
            {
                throw new ArgumentException($"Failed to read the vocabulary file.");
            }

            return vocab;
        }

        internal static Dictionary<StringSpanOrdinalKeyPair, int> GetMergeRanks(Stream mergeStream)
        {
            var mergeRanks = new Dictionary<StringSpanOrdinalKeyPair, int>();
            try
            {
                using StreamReader reader = new StreamReader(mergeStream);

                // We ignore the first and last line in the file

View on GitHub (pinned to 7b76e69cf9)