{"record":{"id":"633c06a626a2cee3","repo":"dotnet/machinelearning","slug":"the-tokenizer-json-does-not-contain-the-required-s","errorCode":null,"errorMessage":"The tokenizer.json does not contain the required special token '{tokenName}'.","messagePattern":"The tokenizer\\.json does not contain the required special token '(.+?)'\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs","lineNumber":1002,"sourceCode":"                target.Add((id, token));\n            }\n        }\n\n        private static void AddAffixToken(\n            List<(int Id, string Token)> target,\n            string tokenName,\n            IReadOnlyList<(string Piece, float Score)> vocab,\n            IReadOnlyDictionary<string, int> specialTokens,\n            bool required)\n        {\n            int id = specialTokens.TryGetValue(tokenName, out int specialId) ? specialId : FindPieceId(vocab, tokenName);\n            if (id >= 0)\n            {\n                target.Add((id, tokenName));\n            }\n            else if (required)\n            {\n                throw new InvalidDataException($\"The tokenizer.json does not contain the required special token '{tokenName}'.\");\n            }\n        }\n\n        private static int FindPieceId(IReadOnlyList<(string Piece, float Score)> vocab, string token)\n        {\n            for (int i = 0; i < vocab.Count; i++)\n            {\n                if (vocab[i].Piece == token)\n                {\n                    return i;\n                }\n            }\n\n            return -1;\n        }\n\n        // Reads a string-valued property, returning null when the property is absent or not a JSON string. Keeps the\n        // switch/compare parsing paths from throwing InvalidOperationException on malformed (non-string) tokenizer.json.","sourceCodeStart":984,"sourceCodeEnd":1020,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs#L984-L1020","documentation":"AddAffixToken looks up a required post-processor affix token (e.g. BOS/EOS for the Sequence template) in the vocabulary/added tokens; if the id is not found (-1) and the token was marked required, InvalidDataException is thrown because the processor cannot build its prefix/suffix.","triggerScenarios":"ResolvePostProcessorAffixes requests a mandatory special token (for example \"<s>\" or \"</s>\") that is absent from both the tokenizer.json added_tokens and the vocabulary — e.g. a Metaspace post_processor template referencing EOS in a model that never defined it.","commonSituations":"Stripped or pruned tokenizers with special tokens removed; using a base-model tokenizer.json where the post-processor expects tokens only present in fine-tuned variants; typos in template token names.","solutions":["Add the missing special token to tokenizer.json's added_tokens (and model vocab) with a valid id, or re-export the tokenizer","Change the post_processor template to reference tokens that exist in this vocabulary","Set the affix as optional / remove it from the template if the token is genuinely not needed","Check that the downloaded tokenizer.json matches the model card (unstripped version)"],"exampleFix":"// before (added_tokens lacks \"</s>\")\n{\"added_tokens\": [{\"content\": \"<s>\", \"id\": 1}]}\n// after\n{\"added_tokens\": [{\"content\": \"<s>\", \"id\": 1}, {\"content\": \"</s>\", \"id\": 2}]}","handlingStrategy":"validation","validationCode":"int id = addedTokens.TryGetValue(name, out var a) ? a\n        : vocab.IndexOf(name);\nif (id < 0 && required) throw new InvalidOperationException($\"Missing required token '{name}'\");","typeGuard":"static bool RequiredTokenPresent(string name, IReadOnlyDictionary<string,int> added, IReadOnlyList<string> vocab) =>\n    added.ContainsKey(name) || vocab.Contains(name);","tryCatchPattern":"try { tok = SentencePieceTokenizer.Create(...); }\ncatch (InvalidDataException ex) when (ex.Message.Contains(\"required special token\"))\n{ /* use the unstripped tokenizer.json or add the token */ }","preventionTips":["Verify required BOS/EOS tokens exist before loading pruned tokenizers","Download tokenizer.json from the model card link, not third-party copies","List added_tokens during onboarding of a new tokenizer"],"tags":["tokenizer","special-token","not-found","affix"],"backgroundTag":"resource-not-found","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}