{"record":{"id":"da7a03a777c13f3a","repo":"dotnet/machinelearning","slug":"the-model-modelname-is-not-supported","errorCode":null,"errorMessage":"The model '{modelName}' is not supported.","messagePattern":"The model '(.+?)' is not supported\\.","errorType":"exception","errorClass":"NotSupportedException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs","lineNumber":1171,"sourceCode":"                                                            };\n\n        private static ModelEncoding GetModelEncoding(string modelName)\n        {\n            if (!_modelToEncoding.TryGetValue(modelName, out ModelEncoding encoder))\n            {\n                foreach ((string Prefix, ModelEncoding Encoding) in _modelPrefixToEncoding)\n                {\n                    if (modelName.StartsWith(Prefix, StringComparison.OrdinalIgnoreCase))\n                    {\n                        encoder = Encoding;\n                        break;\n                    }\n                }\n            }\n\n            if (encoder == ModelEncoding.None)\n            {\n                throw new NotSupportedException($\"The model '{modelName}' is not supported.\");\n            }\n\n            return encoder;\n        }\n\n        private static Dictionary<string, int> CreateHarmonyEncodingSpecialTokens() =>\n            new Dictionary<string, int>\n            {\n                { StartOfText,                  199998 },\n                { EndOfText,                    199999 },\n                { $\"{ReservedPrefix}200000|>\",  200000 },\n                { $\"{ReservedPrefix}200001|>\",  200001 },\n                { Return,                       200002 },\n                { Constrain,                    200003 },\n                { $\"{ReservedPrefix}200004|>\",  200004 },\n                { Channel,                      200005 },\n                { Start,                        200006 },\n                { End,                          200007 },","sourceCodeStart":1153,"sourceCodeEnd":1189,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs#L1153-L1189","documentation":"TiktokenTokenizer.CreateForModel resolves the model name to a ModelEncoding; if the name matches no supported OpenAI model (e.g. a new or third-party model), a NotSupportedException 'The model X is not supported.' is thrown. The library only knows a fixed set of encodings (cl100k_base, p50k_base, o200k_base, etc.).","triggerScenarios":"TiktokenTokenizer.CreateForModel(\"some-new-model\") (or the ctor taking modelName) with a model name added after the installed library version, or a non-OpenAI model name.","commonSituations":"Passing a model released after your Microsoft.ML.Tokenizers version (library not updated); typos in model names; using model names from other providers (Anthropic, Gemini) with a tiktoken tokenizer.","solutions":["Upgrade to the latest Microsoft.ML.Tokenizers and its Data packages, which support newer models.","Use TiktokenTokenizer.CreateForModel with a known name or map your model to the closest known encoding (e.g. o200k_base for GPT-4o family).","Build the tokenizer explicitly with TiktokenTokenizer.CreateAsync(stream, specialTokens) using the matching vocab file.","Normalize/case-check the model name against the supported list before calling."],"exampleFix":"// before\nvar tok = TiktokenTokenizer.CreateForModel(\"gpt-5-turbo-preview-x\"); // not supported\n// after\nvar tok = TiktokenTokenizer.CreateForModel(\"gpt-4o\"); // or explicit encoding\ntok ??= await TiktokenTokenizer.CreateAsync(vocabStream, new Dictionary<string,int> { { TiktokenTokenizer.EndOfText, 50256 } });","handlingStrategy":"fallback","validationCode":"static readonly HashSet<string> KnownModels = new(StringComparer.OrdinalIgnoreCase)\n{ \"gpt2\", \"gpt-2\", \"gpt-3.5-turbo\", \"gpt-4\", \"gpt-4o\", \"gpt-4o-mini\", \"text-embedding-ada-002\", \"text-davinci-003\" /* match your library version's list */ };\nif (!KnownModels.Contains(modelName)) throw new NotSupportedException($\"Model '{modelName}' not supported by this Tokenizers version\");","typeGuard":"static bool IsSupportedModel(string? modelName) =>\n    modelName is not null && TiktokenTokenizer.CreateForModel(modelName) is not null; // or maintain a static allow-list","tryCatchPattern":"try { tok = TiktokenTokenizer.CreateForModel(modelName); }\ncatch (NotSupportedException)\n{ tok = await TiktokenTokenizer.CreateAsync(o200kVocabStream, new Dictionary<string,int> { { TiktokenTokenizer.EndOfText, 50256 } }); } // explicit vocab fallback","preventionTips":["Keep Microsoft.ML.Tokenizers and its Data packages up to date to support new models.","Only pass OpenAI model names to CreateForModel; use explicit vocab streams for other providers.","Maintain a model->encoding map in your app so new/unknown names fall back predictably."],"tags":["tokenizer","not-supported","unsupported-model","tiktoken"],"backgroundTag":"unsupported-enum-value","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}