{"record":{"id":"5fc53119561eb811","repo":"dotnet/machinelearning","slug":"the-vocabulary-does-not-contain-the-required-speci","errorCode":null,"errorMessage":"The vocabulary does not contain the required special token.","messagePattern":"The vocabulary does not contain the required special token\\.","errorType":"exception","errorClass":"ArgumentException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceUnigramModel.cs","lineNumber":366,"sourceCode":"                return -1;\n            }\n\n            for (int i = 0; i < pieces.Count; i++)\n            {\n                if (pieces[i].Piece == tokenName)\n                {\n                    return i;\n                }\n            }\n\n            return -1;\n        }\n\n        private static int CheckSpecialId(bool required, int id, string paramName)\n        {\n            if (required && id < 0)\n            {\n                throw new ArgumentException($\"The vocabulary does not contain the required special token.\", paramName);\n            }\n            return id;\n        }\n\n        public override IReadOnlyDictionary<string, int> Vocabulary => new ReadOnlyDictionary<string, int>(_vocab);\n\n        public int MaxIdByteFallbackId { get; }\n\n        public override IReadOnlyList<EncodedToken> EncodeToTokens(string? text, ReadOnlySpan<char> textSpan, out string? normalizedText, bool addBeginningOfSentence, bool addEndOfSentence, bool considerNormalization)\n        {\n            ReadOnlySpan<char> textToEncode = string.IsNullOrEmpty(text) ? textSpan : text.AsSpan();\n            if (textToEncode.IsEmpty)\n            {\n                normalizedText = string.Empty;\n                return Array.Empty<EncodedToken>();\n            }\n\n            List<EncodedToken> tokens = new();","sourceCodeStart":348,"sourceCodeEnd":384,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceUnigramModel.cs#L348-L384","documentation":"CheckSpecialId validates special-token ids (bos, eos, etc.): when the token is marked required and its id is -1 (meaning 'not present'), it throws ArgumentException with the message 'The vocabulary does not contain the required special token.' The library requires that a mandatory special token actually exist in the vocabulary.","triggerScenarios":"Constructing SentencePieceUnigramModel where a required special token (e.g. bos_id or eos_id per the configuration) is -1/absent in the model's trainer spec or id lookup.","commonSituations":"Models trained without BOS/EOS tokens but loaded with options marking them required; tokenizer.json configs omitting bos_token/eos_token; manually stripped special tokens from the vocabulary.","solutions":["Provide a valid id for the required special token in the model configuration (tokenizer.json bos_token/eos_token or proto TrainerSpec).","If the model truly has no such token, configure it as optional instead of required so -1 is accepted.","Retrain or re-export the SentencePiece model with the needed special tokens defined.","Pre-check with Python sentencepiece: confirm sp.bos_id()/sp.eos_id() are not -1 before loading in .NET."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// C# — ensure required special tokens exist before loading\nforeach (var (name, id) in new[] { (\"bos\", bosId), (\"eos\", eosId) })\n{\n    if (requireSpecialTokens && id < 0)\n        throw new ArgumentException($\"Required special token '{name}' not present in vocabulary.\");\n}","typeGuard":"static bool SpecialTokenPresent(bool required, int id) => !required || id >= 0;","tryCatchPattern":"try { var model = new SentencePieceUnigramModel(modelProto); }\ncatch (ArgumentException ex) when (ex.Message.Contains(\"special token\")) { /* model lacks required bos/eos — retrain or mark optional */ }","preventionTips":["Train SentencePiece models with --bos_id/--eos_id set when downstream code requires them.","Mark special tokens optional in config if the model was trained without them.","Check sp.bos_id()/sp.eos_id() != -1 in Python before deploying the model to .NET.","Keep the set of required special tokens in sync with the model training configuration."],"tags":["sentencepiece","special-tokens","vocabulary","model-loading"],"backgroundTag":"missing-required-argument","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}