dotnet/machinelearning · error · ArgumentException

The encoding name '{encodingName}' is not supported. The onl

Error message

The encoding name '{encodingName}' is not supported. The only supported encoding names are: {TiktokenTokenizer.Cl100kBaseEncodingName}, {TiktokenTokenizer.P50kBaseEncodingName}, {TiktokenTokenizer.P50kEditEncodingName}, and {TiktokenTokenizer.R50kBaseEncodingName}.

What it means

TiktokenTokenizer.CreateForModelAndGetEncoding / CreateForModel only accept four known encoding names (cl100k_base, p50k_base, p50k_edit, r50k_base). Passing any other string falls into the else branch and throws ArgumentException naming the offending encodingName. The comparison is OrdinalIgnoreCase, so casing is never the cause — only an unknown encoding name is.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs:1573

            else if (encodingName.Equals(O200kHarmonyEncodingName, StringComparison.OrdinalIgnoreCase))
            {
                modelEncoding = ModelEncoding.O200kHarmony;
            }
            else if (encodingName.Equals(P50kBaseEncodingName, StringComparison.OrdinalIgnoreCase))
            {
                modelEncoding = ModelEncoding.P50kBase;
            }
            else if (encodingName.Equals(P50kEditEncodingName, StringComparison.OrdinalIgnoreCase))
            {
                modelEncoding = ModelEncoding.P50kEdit;
            }
            else if (encodingName.Equals(R50kBaseEncodingName, StringComparison.OrdinalIgnoreCase))
            {
                modelEncoding = ModelEncoding.R50kBase;
            }
            else
            {
                throw new ArgumentException($"The encoding name '{encodingName}' is not supported. The only supported encoding names are: {TiktokenTokenizer.Cl100kBaseEncodingName}, {TiktokenTokenizer.P50kBaseEncodingName}, {TiktokenTokenizer.P50kEditEncodingName}, and {TiktokenTokenizer.R50kBaseEncodingName}.", nameof(encodingName));
            }

            return CreateForModel(modelEncoding, modelName: null, extraSpecialTokens, normalizer);
        }
    }
}

View on GitHub (pinned to 7b76e69cf9)

Solutions

Use one of the supported names: cl100k_base, p50k_base, p50k_edit, or r50k_base (case-insensitive)
If you need a newer encoding like o200k_base, upgrade Microsoft.ML.Tokenizers to a version that supports it, or use a different tokenizer package (e.g. SharpToken/Tiktoken nuget)
Check the constant properties TiktokenTokenizer.Cl100kBaseEncodingName etc. instead of hardcoding strings

Example fix

// before
var tok = TiktokenTokenizer.CreateForModel("o200k_base");
// after
var tok = TiktokenTokenizer.CreateForModel(TiktokenTokenizer.Cl100kBaseEncodingName);

Defensive patterns

Strategy: validation

Validate before calling

string[] supported = { TiktokenTokenizer.Cl100kBaseEncodingName, TiktokenTokenizer.P50kBaseEncodingName, TiktokenTokenizer.P50kEditEncodingName, TiktokenTokenizer.R50kBaseEncodingName };
if (!supported.Contains(encodingName, StringComparer.OrdinalIgnoreCase)) throw new ArgumentException($"Unsupported encoding '{encodingName}'. Supported: {string.Join(", ", supported)}");

Type guard

bool IsSupportedEncoding(string? name) => name is not null && (name.Equals(TiktokenTokenizer.Cl100kBaseEncodingName, StringComparison.OrdinalIgnoreCase) || name.Equals(TiktokenTokenizer.P50kBaseEncodingName, StringComparison.OrdinalIgnoreCase) || name.Equals(TiktokenTokenizer.P50kEditEncodingName, StringComparison.OrdinalIgnoreCase) || name.Equals(TiktokenTokenizer.R50kBaseEncodingName, StringComparison.OrdinalIgnoreCase));

Try / catch

try { var tok = TiktokenTokenizer.CreateForModel(encodingName); } catch (ArgumentException ex) when (ex.ParamName == "encodingName") { log.LogError(ex, "Unsupported tiktoken encoding"); /* fallback to Cl100kBase or surface config error */ }

Prevention

Reference the static encoding-name constants instead of raw strings
Check library release notes for newly supported encodings before targeting new models
Centralize tokenizer creation in one factory method with validation

When it happens

Trigger: Calling TiktokenTokenizer.CreateForModel (or the Create overload that takes an encodingName) with a string other than the four supported names, e.g. "o200k_base", "gpt2", "text-davinci-003", or a typo like "cl100k-bas".

Common situations: Using a tokenizer for a newer OpenAI model (o200k_base for GPT-4o) whose encoding this library version doesn't support yet; passing a model name instead of an encoding name; porting Python tiktoken code where arbitrary encodings exist.

Understand the failure class

Background: Invalid enum value errors: "Unknown type", "Invalid scope", "must be one of" — when a string is not on the library's allowed list — this error's family across 23 libraries.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/3f3e222cbc6eae3b. Report an issue: GitHub.