dotnet/machinelearning · error · InvalidDataException

An 'added_tokens' entry must have a string 'content' and a n

Error message

An 'added_tokens' entry must have a string 'content' and a numeric 'id'.

What it means

ParseAddedTokens reads the 'added_tokens' array from tokenizer.json and requires each entry to have a string 'content' and a numeric 'id' so it can build the token-to-id map. When an entry is missing either property, or either property has the wrong JSON type, the tokenizer cannot key the map and throws InvalidDataException rather than silently producing a broken vocabulary.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs:786

            foreach (JsonElement entry in addedTokens.EnumerateArray())
            {
                if (entry.ValueKind != JsonValueKind.Object)
                {
                    continue;
                }

                if (!entry.TryGetProperty("special", out JsonElement specialElement) || specialElement.ValueKind != JsonValueKind.True)
                {
                    continue;
                }

                if (entry.TryGetProperty("content", out JsonElement contentElement) &&
                    entry.TryGetProperty("id", out JsonElement idElement))
                {
                    if (contentElement.ValueKind != JsonValueKind.String || idElement.ValueKind != JsonValueKind.Number)
                    {
                        throw new InvalidDataException("An 'added_tokens' entry must have a string 'content' and a numeric 'id'.");
                    }

                    result[contentElement.GetString()!] = idElement.GetInt32();
                }
            }

            return result;
        }

        // Resolves the ordered prefix/suffix special tokens that wrap an encoded sequence, from the post_processor.
        private static void ResolvePostProcessorAffixes(
            JsonElement root,
            IReadOnlyList<(string Piece, float Score)> vocab,
            IReadOnlyDictionary<string, int> specialTokens,
            out List<(int Id, string Token)> prefixTokens,
            out List<(int Id, string Token)> suffixTokens)
        {
            prefixTokens = new List<(int Id, string Token)>();

View on GitHub (pinned to 7b76e69cf9)

Solutions

Open tokenizer.json and ensure every added_tokens entry has exactly {"content": "<string>", "id": <number>}
Regenerate tokenizer.json with the HuggingFace tokenizers library (tokenizer.save / save_pretrained) instead of editing by hand
Re-download the tokenizer files from the model repository; the file may be corrupted or truncated
Pre-validate the JSON: parse added_tokens and check entry.TryGetProperty("content")/("id") and their ValueKinds before constructing the tokenizer

Example fix

// before (tokenizer.json)
{"added_tokens": [{"content": "<s>", "id": "1"}]}
// after
{"added_tokens": [{"content": "<s>", "id": 1}]}

Defensive patterns

Strategy: validation

Validate before calling

using var doc = JsonDocument.Parse(File.ReadAllText(path));
foreach (var e in doc.RootElement.GetProperty("added_tokens").EnumerateArray())
    if (e.ValueKind != JsonValueKind.Object ||
        !e.TryGetProperty("content", out var c) || c.ValueKind != JsonValueKind.String ||
        !e.TryGetProperty("id", out var i) || i.ValueKind != JsonValueKind.Number)
        throw new InvalidDataException("Bad added_tokens entry");

Type guard

static bool IsValidAddedToken(JsonElement e) =>
    e.ValueKind == JsonValueKind.Object &&
    e.TryGetProperty("content", out var c) && c.ValueKind == JsonValueKind.String &&
    e.TryGetProperty("id", out var i) && i.ValueKind == JsonValueKind.Number;

Try / catch

try { tokenizer = SentencePieceTokenizer.Create(modelStream, vocabStream); }
catch (InvalidDataException ex) when (ex.Message.Contains("added_tokens"))
{ /* repair or re-download tokenizer.json */ }

Prevention

Never hand-edit tokenizer.json; regenerate with HuggingFace save_pretrained
Validate tokenizer.json against the HuggingFace tokenizers JSON schema before loading
Verify downloaded files (size/checksum) to catch truncation
Keep tokenizer.json, vocab, and model files from the same model revision

When it happens

Trigger: Loading a tokenizer.json whose added_tokens entry lacks 'content' or 'id', has them under different casing/names, has content as a number or id as a string (e.g. "id": "0"), or is a non-object element like a string. Raised from ParseAddedTokens while SentencePieceTokenizer.FromTokenizerJson / mergedSpecialTokens builds its special-token table.

Common situations: Hand-edited tokenizer.json files; tokenizers produced by non-HuggingFace tools or older/other libraries that serialize ids as strings; truncated or corrupted downloads of tokenizer.json; schemas where added_tokens use 'id_str' or 'token' field names.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/5392c4d62f8efa57. Report an issue: GitHub.