{"record":{"id":"5392c4d62f8efa57","repo":"dotnet/machinelearning","slug":"an-added-tokens-entry-must-have-a-string-conten","errorCode":null,"errorMessage":"An 'added_tokens' entry must have a string 'content' and a numeric 'id'.","messagePattern":"An 'added_tokens' entry must have a string 'content' and a numeric 'id'\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs","lineNumber":786,"sourceCode":"\n            foreach (JsonElement entry in addedTokens.EnumerateArray())\n            {\n                if (entry.ValueKind != JsonValueKind.Object)\n                {\n                    continue;\n                }\n\n                if (!entry.TryGetProperty(\"special\", out JsonElement specialElement) || specialElement.ValueKind != JsonValueKind.True)\n                {\n                    continue;\n                }\n\n                if (entry.TryGetProperty(\"content\", out JsonElement contentElement) &&\n                    entry.TryGetProperty(\"id\", out JsonElement idElement))\n                {\n                    if (contentElement.ValueKind != JsonValueKind.String || idElement.ValueKind != JsonValueKind.Number)\n                    {\n                        throw new InvalidDataException(\"An 'added_tokens' entry must have a string 'content' and a numeric 'id'.\");\n                    }\n\n                    result[contentElement.GetString()!] = idElement.GetInt32();\n                }\n            }\n\n            return result;\n        }\n\n        // Resolves the ordered prefix/suffix special tokens that wrap an encoded sequence, from the post_processor.\n        private static void ResolvePostProcessorAffixes(\n            JsonElement root,\n            IReadOnlyList<(string Piece, float Score)> vocab,\n            IReadOnlyDictionary<string, int> specialTokens,\n            out List<(int Id, string Token)> prefixTokens,\n            out List<(int Id, string Token)> suffixTokens)\n        {\n            prefixTokens = new List<(int Id, string Token)>();","sourceCodeStart":768,"sourceCodeEnd":804,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs#L768-L804","documentation":"ParseAddedTokens reads the 'added_tokens' array from tokenizer.json and requires each entry to have a string 'content' and a numeric 'id' so it can build the token-to-id map. When an entry is missing either property, or either property has the wrong JSON type, the tokenizer cannot key the map and throws InvalidDataException rather than silently producing a broken vocabulary.","triggerScenarios":"Loading a tokenizer.json whose added_tokens entry lacks 'content' or 'id', has them under different casing/names, has content as a number or id as a string (e.g. \"id\": \"0\"), or is a non-object element like a string. Raised from ParseAddedTokens while SentencePieceTokenizer.FromTokenizerJson / mergedSpecialTokens builds its special-token table.","commonSituations":"Hand-edited tokenizer.json files; tokenizers produced by non-HuggingFace tools or older/other libraries that serialize ids as strings; truncated or corrupted downloads of tokenizer.json; schemas where added_tokens use 'id_str' or 'token' field names.","solutions":["Open tokenizer.json and ensure every added_tokens entry has exactly {\"content\": \"<string>\", \"id\": <number>}","Regenerate tokenizer.json with the HuggingFace tokenizers library (tokenizer.save / save_pretrained) instead of editing by hand","Re-download the tokenizer files from the model repository; the file may be corrupted or truncated","Pre-validate the JSON: parse added_tokens and check entry.TryGetProperty(\"content\")/(\"id\") and their ValueKinds before constructing the tokenizer"],"exampleFix":"// before (tokenizer.json)\n{\"added_tokens\": [{\"content\": \"<s>\", \"id\": \"1\"}]}\n// after\n{\"added_tokens\": [{\"content\": \"<s>\", \"id\": 1}]}","handlingStrategy":"validation","validationCode":"using var doc = JsonDocument.Parse(File.ReadAllText(path));\nforeach (var e in doc.RootElement.GetProperty(\"added_tokens\").EnumerateArray())\n    if (e.ValueKind != JsonValueKind.Object ||\n        !e.TryGetProperty(\"content\", out var c) || c.ValueKind != JsonValueKind.String ||\n        !e.TryGetProperty(\"id\", out var i) || i.ValueKind != JsonValueKind.Number)\n        throw new InvalidDataException(\"Bad added_tokens entry\");","typeGuard":"static bool IsValidAddedToken(JsonElement e) =>\n    e.ValueKind == JsonValueKind.Object &&\n    e.TryGetProperty(\"content\", out var c) && c.ValueKind == JsonValueKind.String &&\n    e.TryGetProperty(\"id\", out var i) && i.ValueKind == JsonValueKind.Number;","tryCatchPattern":"try { tokenizer = SentencePieceTokenizer.Create(modelStream, vocabStream); }\ncatch (InvalidDataException ex) when (ex.Message.Contains(\"added_tokens\"))\n{ /* repair or re-download tokenizer.json */ }","preventionTips":["Never hand-edit tokenizer.json; regenerate with HuggingFace save_pretrained","Validate tokenizer.json against the HuggingFace tokenizers JSON schema before loading","Verify downloaded files (size/checksum) to catch truncation","Keep tokenizer.json, vocab, and model files from the same model revision"],"tags":["json","tokenizer","schema-validation","invalid-data"],"backgroundTag":"schema-validation-failed","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}