dotnet/machinelearning · error · JsonException

Invalid JSON.

Error message

Invalid JSON.

What it means

This JsonException is the fall-through failure of a custom Vocabulary JsonConverter's Read method: it reads a JSON object of string->int mappings and throws 'Invalid JSON.' whenever the structure doesn't match (non-object token, non-integer value, missing properties, or the reader never reaching EndObject). It exists to give a consistent failure when the serialized vocabulary doesn't match the expected shape.

Source

Thrown at src/Microsoft.ML.Tokenizers/Utils/StringSpanOrdinalKey.cs:166

        public override Vocabulary Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
        {
            var dictionary = new Vocabulary();
            while (reader.Read())
            {
                if (reader.TokenType == JsonTokenType.EndObject)
                {
                    return dictionary;
                }

                if (reader.TokenType == JsonTokenType.PropertyName)
                {
                    var key = reader.GetString();
                    reader.Read();
                    var value = reader.GetInt32();
                    dictionary.Add(new StringSpanOrdinalKey(key!), (value, key!));
                }
            }
            throw new JsonException("Invalid JSON.");
        }

        public override void Write(Utf8JsonWriter writer, Vocabulary value, JsonSerializerOptions options) => throw new NotImplementedException();
    }

    /// <summary>
    /// Extension methods for <see cref="StringSpanOrdinalKey"/>.
    /// </summary>
    internal static class StringSpanOrdinalKeyExtensions
    {
        public static unsafe bool TryGetValue<TValue>(this Dictionary<StringSpanOrdinalKey, TValue> map, ReadOnlySpan<char> key, out TValue value)
        {
            fixed (char* ptr = key)
            {
                return map.TryGetValue(new StringSpanOrdinalKey(ptr, key.Length), out value!);
            }
        }

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Check the JSON input is a flat {"token": id, ...} object with integer values; fix or regenerate the file.
2. Compare against the format produced by the same library version's Write/serialization path.
3. Deserialize from the original vocab source rather than a hand-edited copy.
4. Catch JsonException and inspect reader.TokenType/position to pinpoint the offending JSON location.

Example fix

// before: values as strings
{ "hello": "12", "world": "34" }
// after
{ "hello": 12, "world": 34 }

using var doc = JsonDocument.Parse(json);
bool validShape = doc.RootElement.ValueKind == JsonValueKind.Object && doc.RootElement.EnumerateObject().All(p => p.Value.ValueKind == JsonValueKind.Number);

static bool LooksLikeVocabulary(string json) { try { using var d = JsonDocument.Parse(json); return d.RootElement.ValueKind == JsonValueKind.Object; } catch (JsonException) { return false; } }

try { vocab = JsonSerializer.Deserialize<Vocabulary>(json, options); }
catch (JsonException ex)
{ throw new InvalidDataException("Vocabulary JSON is not a string->int object", ex); }

When it happens

Trigger: Deserializing a Vocabulary from JSON that is not a flat object of string keys to integer values — e.g. an array instead of an object, nested values, string values for IDs, or truncated JSON.

Common situations: Loading a vocab.json saved in a different format/version than the converter expects, hand-editing the vocab file and breaking the schema, or passing a non-vocab JSON stream to the deserializer.

Understand the failure class

Background: JSON parse error: "Unexpected token" / "not valid JSON" / "failed to parse" — what JSON parsers are really complaining about — this error's family across 45 libraries.

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• Problems met when parsing JSON vocabulary object.{Environmen
• unknown schema type: {schema.Type}
• unknown option type
• Unsupported reader type {reader.TokenType}
• Problems met when parsing JSON vocabulary object.{Environmen

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/e2fd2d25b6016191. Report an issue: GitHub.