dotnet/machinelearning · error · NotSupportedException

The model ' ' is not supported.

Error message

The model '{modelName}' is not supported.

What it means

TiktokenTokenizer.CreateForModel resolves the model name to a ModelEncoding; if the name matches no supported OpenAI model (e.g. a new or third-party model), a NotSupportedException 'The model X is not supported.' is thrown. The library only knows a fixed set of encodings (cl100k_base, p50k_base, o200k_base, etc.).

Solutions

1. Upgrade to the latest Microsoft.ML.Tokenizers and its Data packages, which support newer models.
2. Use TiktokenTokenizer.CreateForModel with a known name or map your model to the closest known encoding (e.g. o200k_base for GPT-4o family).
3. Build the tokenizer explicitly with TiktokenTokenizer.CreateAsync(stream, specialTokens) using the matching vocab file.
4. Normalize/case-check the model name against the supported list before calling.

Example fix

// before
var tok = TiktokenTokenizer.CreateForModel("gpt-5-turbo-preview-x"); // not supported
// after
var tok = TiktokenTokenizer.CreateForModel("gpt-4o"); // or explicit encoding
tok ??= await TiktokenTokenizer.CreateAsync(vocabStream, new Dictionary<string,int> { { TiktokenTokenizer.EndOfText, 50256 } });

static readonly HashSet<string> KnownModels = new(StringComparer.OrdinalIgnoreCase)
{ "gpt2", "gpt-2", "gpt-3.5-turbo", "gpt-4", "gpt-4o", "gpt-4o-mini", "text-embedding-ada-002", "text-davinci-003" /* match your library version's list */ };
if (!KnownModels.Contains(modelName)) throw new NotSupportedException($"Model '{modelName}' not supported by this Tokenizers version");

static bool IsSupportedModel(string? modelName) =>
    modelName is not null && TiktokenTokenizer.CreateForModel(modelName) is not null; // or maintain a static allow-list

try { tok = TiktokenTokenizer.CreateForModel(modelName); }
catch (NotSupportedException)
{ tok = await TiktokenTokenizer.CreateAsync(o200kVocabStream, new Dictionary<string,int> { { TiktokenTokenizer.EndOfText, 50256 } }); } // explicit vocab fallback

When it happens

Trigger: TiktokenTokenizer.CreateForModel("some-new-model") (or the ctor taking modelName) with a model name added after the installed library version, or a non-OpenAI model name.

Common situations: Passing a model released after your Microsoft.ML.Tokenizers version (library not updated); typos in model names; using model names from other providers (Anthropic, Gemini) with a tiktoken tokenizer.

Related errors

• The model ' ' is not supported.
• A piece string in 'model.vocab' is null.
• A post_processor template 'SpecialToken.id' must be a…
• An 'added_tokens' entry must have a string 'content' and a…
• argument should not be null.

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/da7a03a777c13f3a. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs:1171

                                                            };

        private static ModelEncoding GetModelEncoding(string modelName)
        {
            if (!_modelToEncoding.TryGetValue(modelName, out ModelEncoding encoder))
            {
                foreach ((string Prefix, ModelEncoding Encoding) in _modelPrefixToEncoding)
                {
                    if (modelName.StartsWith(Prefix, StringComparison.OrdinalIgnoreCase))
                    {
                        encoder = Encoding;
                        break;
                    }
                }
            }

            if (encoder == ModelEncoding.None)
            {
                throw new NotSupportedException($"The model '{modelName}' is not supported.");
            }

            return encoder;
        }

        private static Dictionary<string, int> CreateHarmonyEncodingSpecialTokens() =>
            new Dictionary<string, int>
            {
                { StartOfText,                  199998 },
                { EndOfText,                    199999 },
                { $"{ReservedPrefix}200000|>",  200000 },
                { $"{ReservedPrefix}200001|>",  200001 },
                { Return,                       200002 },
                { Constrain,                    200003 },
                { $"{ReservedPrefix}200004|>",  200004 },
                { Channel,                      200005 },
                { Start,                        200006 },
                { End,                          200007 },

View on GitHub (pinned to 7b76e69cf9)