{"record":{"id":"3f3e222cbc6eae3b","repo":"dotnet/machinelearning","slug":"the-encoding-name-encodingname-is-not-supporte","errorCode":null,"errorMessage":"The encoding name '{encodingName}' is not supported. The only supported encoding names are: {TiktokenTokenizer.Cl100kBaseEncodingName}, {TiktokenTokenizer.P50kBaseEncodingName}, {TiktokenTokenizer.P50kEditEncodingName}, and {TiktokenTokenizer.R50kBaseEncodingName}.","messagePattern":"The encoding name '(.+?)' is not supported\\. The only supported encoding names are: (.+?), (.+?), (.+?), and (.+?)\\.","errorType":"exception","errorClass":"ArgumentException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs","lineNumber":1573,"sourceCode":"            else if (encodingName.Equals(O200kHarmonyEncodingName, StringComparison.OrdinalIgnoreCase))\n            {\n                modelEncoding = ModelEncoding.O200kHarmony;\n            }\n            else if (encodingName.Equals(P50kBaseEncodingName, StringComparison.OrdinalIgnoreCase))\n            {\n                modelEncoding = ModelEncoding.P50kBase;\n            }\n            else if (encodingName.Equals(P50kEditEncodingName, StringComparison.OrdinalIgnoreCase))\n            {\n                modelEncoding = ModelEncoding.P50kEdit;\n            }\n            else if (encodingName.Equals(R50kBaseEncodingName, StringComparison.OrdinalIgnoreCase))\n            {\n                modelEncoding = ModelEncoding.R50kBase;\n            }\n            else\n            {\n                throw new ArgumentException($\"The encoding name '{encodingName}' is not supported. The only supported encoding names are: {TiktokenTokenizer.Cl100kBaseEncodingName}, {TiktokenTokenizer.P50kBaseEncodingName}, {TiktokenTokenizer.P50kEditEncodingName}, and {TiktokenTokenizer.R50kBaseEncodingName}.\", nameof(encodingName));\n            }\n\n            return CreateForModel(modelEncoding, modelName: null, extraSpecialTokens, normalizer);\n        }\n    }\n}\n","sourceCodeStart":1555,"sourceCodeEnd":1580,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs#L1555-L1580","documentation":"TiktokenTokenizer.CreateForModelAndGetEncoding / CreateForModel only accept four known encoding names (cl100k_base, p50k_base, p50k_edit, r50k_base). Passing any other string falls into the else branch and throws ArgumentException naming the offending encodingName. The comparison is OrdinalIgnoreCase, so casing is never the cause — only an unknown encoding name is.","triggerScenarios":"Calling TiktokenTokenizer.CreateForModel (or the Create overload that takes an encodingName) with a string other than the four supported names, e.g. \"o200k_base\", \"gpt2\", \"text-davinci-003\", or a typo like \"cl100k-bas\".","commonSituations":"Using a tokenizer for a newer OpenAI model (o200k_base for GPT-4o) whose encoding this library version doesn't support yet; passing a model name instead of an encoding name; porting Python tiktoken code where arbitrary encodings exist.","solutions":["Use one of the supported names: cl100k_base, p50k_base, p50k_edit, or r50k_base (case-insensitive)","If you need a newer encoding like o200k_base, upgrade Microsoft.ML.Tokenizers to a version that supports it, or use a different tokenizer package (e.g. SharpToken/Tiktoken nuget)","Check the constant properties TiktokenTokenizer.Cl100kBaseEncodingName etc. instead of hardcoding strings"],"exampleFix":"// before\nvar tok = TiktokenTokenizer.CreateForModel(\"o200k_base\");\n// after\nvar tok = TiktokenTokenizer.CreateForModel(TiktokenTokenizer.Cl100kBaseEncodingName);","handlingStrategy":"validation","validationCode":"string[] supported = { TiktokenTokenizer.Cl100kBaseEncodingName, TiktokenTokenizer.P50kBaseEncodingName, TiktokenTokenizer.P50kEditEncodingName, TiktokenTokenizer.R50kBaseEncodingName };\nif (!supported.Contains(encodingName, StringComparer.OrdinalIgnoreCase)) throw new ArgumentException($\"Unsupported encoding '{encodingName}'. Supported: {string.Join(\", \", supported)}\");","typeGuard":"bool IsSupportedEncoding(string? name) => name is not null && (name.Equals(TiktokenTokenizer.Cl100kBaseEncodingName, StringComparison.OrdinalIgnoreCase) || name.Equals(TiktokenTokenizer.P50kBaseEncodingName, StringComparison.OrdinalIgnoreCase) || name.Equals(TiktokenTokenizer.P50kEditEncodingName, StringComparison.OrdinalIgnoreCase) || name.Equals(TiktokenTokenizer.R50kBaseEncodingName, StringComparison.OrdinalIgnoreCase));","tryCatchPattern":"try { var tok = TiktokenTokenizer.CreateForModel(encodingName); } catch (ArgumentException ex) when (ex.ParamName == \"encodingName\") { log.LogError(ex, \"Unsupported tiktoken encoding\"); /* fallback to Cl100kBase or surface config error */ }","preventionTips":["Reference the static encoding-name constants instead of raw strings","Check library release notes for newly supported encodings before targeting new models","Centralize tokenizer creation in one factory method with validation"],"tags":["tokenizers","argument","encoding"],"backgroundTag":"invalid-enum-value","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}