{"record":{"id":"1bdb8b6f4943f8f8","repo":"dotnet/machinelearning","slug":"the-model-modelname-modelencoding-tostring","errorCode":null,"errorMessage":"The model '{modelName ?? modelEncoding.ToString()}' is not supported.","messagePattern":"The model '(.+?)' is not supported\\.","errorType":"exception","errorClass":"NotSupportedException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs","lineNumber":1238,"sourceCode":"\n                case ModelEncoding.O200kBase:\n                    return (new Dictionary<string, int> { { EndOfText, 199999 }, { EndOfPrompt, 200018 } }, O200kBaseRegex(), O200kBaseFile, Type.GetType(O200kBaseTypeName), O200kBasePackageName);\n\n                case ModelEncoding.P50kBase:\n                    return (new Dictionary<string, int> { { EndOfText, 50256 } }, P50kBaseRegex(), P50RanksFile, Type.GetType(P50kBaseTypeName), P50kBasePackageName);\n\n                case ModelEncoding.P50kEdit:\n                    return (new Dictionary<string, int>\n                        { { EndOfText, 50256 }, { FimPrefix, 50281 }, { FimMiddle, 50282 }, { FimSuffix, 50283 } }, P50kBaseRegex(), P50RanksFile, Type.GetType(P50kBaseTypeName), P50kBasePackageName);\n\n                case ModelEncoding.R50kBase:\n                    return (new Dictionary<string, int> { { EndOfText, 50256 } }, P50kBaseRegex(), R50RanksFile, Type.GetType(R50kBaseTypeName), R50kBasePackageName);\n\n                case ModelEncoding.O200kHarmony:\n                    return (CreateHarmonyEncodingSpecialTokens(), O200kBaseRegex(), O200kBaseFile, Type.GetType(O200kBaseTypeName), O200kBasePackageName);\n\n                default:\n                    throw new NotSupportedException($\"The model '{modelName ?? modelEncoding.ToString()}' is not supported.\");\n            }\n        }\n\n        // Regex patterns based on https://github.com/openai/tiktoken/blob/main/tiktoken_ext/openai_public.py\n\n        private const string Cl100kBaseRegexPattern = /*lang=regex*/ @\"'(?i:[sdmt]|ll|ve|re)|(?>[^\\r\\n\\p{L}\\p{N}]?)(?>\\p{L}+)|(?>\\p{N}{1,3})| ?(?>[^\\s\\p{L}\\p{N}]+)(?>[\\r\\n]*)|(?>\\s+)$|\\s*[\\r\\n]|\\s+(?!\\S)|\\s\";\n        private const string P50kBaseRegexPattern = /*lang=regex*/ @\"'(?:[sdmt]|ll|ve|re)| ?(?>\\p{L}+)| ?(?>\\p{N}+)| ?(?>[^\\s\\p{L}\\p{N}]+)|(?>\\s+)$|\\s+(?!\\S)|\\s\";\n        private const string O200kBaseRegexPattern = /*lang=regex*/ @\"[^\\r\\n\\p{L}\\p{N}]?[\\p{Lu}\\p{Lt}\\p{Lm}\\p{Lo}\\p{M}]*[\\p{Ll}\\p{Lm}\\p{Lo}\\p{M}]+(?i:'s|'t|'re|'ve|'m|'ll|'d)?|[^\\r\\n\\p{L}\\p{N}]?[\\p{Lu}\\p{Lt}\\p{Lm}\\p{Lo}\\p{M}]+[\\p{Ll}\\p{Lm}\\p{Lo}\\p{M}]*(?i:'s|'t|'re|'ve|'m|'ll|'d)?|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n/]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+\";\n\n        private const string Cl100kBaseVocabFile = \"cl100k_base.tiktoken.deflate\";  // \"https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken\"\n        private const string P50RanksFile = \"p50k_base.tiktoken.deflate\";           // \"https://openaipublic.blob.core.windows.net/encodings/p50k_base.tiktoken\"\n        private const string R50RanksFile = \"r50k_base.tiktoken.deflate\";           // \"https://openaipublic.blob.core.windows.net/encodings/r50k_base.tiktoken\"\n        private const string GPT2File = \"gpt2.tiktoken.deflate\";                    // \"https://openaipublic.blob.core.windows.net/encodings/r50k_base.tiktoken\". Gpt2 is using the same encoding as R50kBase\n        private const string O200kBaseFile = \"o200k_base.tiktoken.deflate\";         // \"https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken\"\n\n        internal const string Cl100kBaseEncodingName = \"cl100k_base\";\n        internal const string P50kBaseEncodingName = \"p50k_base\";\n        internal const string P50kEditEncodingName = \"p50k_edit\";","sourceCodeStart":1220,"sourceCodeEnd":1256,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs#L1220-L1256","documentation":"When no explicit modelName is given, the constructor switches on the ModelEncoding enum; a value outside the known cases (default arm) throws NotSupportedException 'The model X is not supported.' This happens with invalid casts or enum values from a different (older/newer) library version.","triggerScenarios":"TiktokenTokenizer ctor / Create overload with modelEncoding set to an undefined enum value (e.g. (ModelEncoding)999 from config deserialization), or a ModelEncoding defined only in a newer/older assembly version.","commonSituations":"Deserializing a numeric encoding from config produced by a different library version; assembly version mismatch where a Data package references newer enum members; manual enum casting from external input.","solutions":["Validate the ModelEncoding value before constructing (Enum.IsDefined) and reject unknown values at the config boundary.","Align all Microsoft.ML.Tokenizers package versions (core + Data packages) so the enum members match.","Use TiktokenTokenizer.CreateForModel with a string model name, which has broader name-based resolution.","If unknown, fall back to creating the tokenizer from an explicit vocab stream + special tokens."],"exampleFix":"// before\nvar enc = (ModelEncoding)int.Parse(cfg[\"encoding\"]);\nvar tok = TiktokenTokenizer.Create(enc);\n// after\nvar enc = (ModelEncoding)int.Parse(cfg[\"encoding\"]);\nif (!Enum.IsDefined(enc)) throw new InvalidOperationException($\"Unknown encoding: {cfg[\"encoding\"]}\");\nvar tok = TiktokenTokenizer.Create(enc);","handlingStrategy":"type-guard","validationCode":"if (!Enum.IsDefined(modelEncoding))\n    throw new InvalidDataException($\"Unknown ModelEncoding value: {(int)modelEncoding}\");","typeGuard":"static bool IsDefinedEncoding(ModelEncoding e) => Enum.IsDefined(e);","tryCatchPattern":"try { var tok = TiktokenTokenizer.Create(encoding); }\ncatch (NotSupportedException ex)\n{ logger.LogError(ex, \"Unsupported ModelEncoding {V}\", encoding); throw new InvalidDataException(\"Configured encoding is not supported by this library version\", ex); }","preventionTips":["Never cast raw ints/strings to ModelEncoding without Enum.IsDefined/Enum.TryParse.","Keep the core package and Data packages on the same version to avoid enum drift.","Validate encoding values read from config at startup."],"tags":["tokenizer","not-supported","invalid-enum","model-encoding"],"backgroundTag":"invalid-enum-value","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}