dotnet/machinelearning · error · InvalidOperationException

The tokenizer data file

Error message

The tokenizer data file {tiktokenConfiguration.PackageName}.dll could not be loaded. Please reference the package {tiktokenConfiguration.PackageName} in your project.

What it means

TiktokenTokenizer resolves vocab data from an embedded resource inside a data-package assembly (Microsoft.ML.Tokenizers.Data.*). If the cached entry is missing and tiktokenConfiguration.DataType is null — meaning the data-package type/assembly could not be located — an InvalidOperationException telling you to reference the data package is thrown.

Solutions

Add the matching NuGet package: Microsoft.ML.Tokenizers.Data.O200kBase (GPT-4o/o200k), .Cl100kBase (GPT-4/3.5), or .P50kBase/.R50kBase for older models.
Run 'dotnet add package Microsoft.ML.Tokenizers.Data.O200kBase' (or the relevant one) and rebuild so the data DLL is deployed with the app.
If self-hosting, ensure the data package assembly is copied to the output and loadable at runtime (check trimming/AOT settings).
Alternatively supply your own vocab stream via TiktokenTokenizer.CreateAsync(stream, specialTokens) instead of relying on built-in data.

Example fix

// before
dotnet add package Microsoft.ML.Tokenizers
var tok = TiktokenTokenizer.CreateForModel("gpt-4o"); // InvalidOperationException: ...Data.O200kBase.dll could not be loaded
// after
dotnet add package Microsoft.ML.Tokenizers
dotnet add package Microsoft.ML.Tokenizers.Data.O200kBase
var tok = TiktokenTokenizer.CreateForModel("gpt-4o");

Defensive patterns

Strategy: try-catch

Try / catch

try { var tok = TiktokenTokenizer.CreateForModel("gpt-4o"); }
catch (InvalidOperationException ex) when (ex.Message.Contains("could not be loaded"))
{
    throw new InvalidOperationException(
        "Missing tokenizer data package. Add e.g. Microsoft.ML.Tokenizers.Data.O200kBase to your project.", ex);
}

Prevention

Reference the required data package (Data.O200kBase / Data.Cl100kBase / Data.P50kBase / Data.R50kBase) whenever you upgrade to Microsoft.ML.Tokenizers 1.0+.
Verify the data DLL ships with your deployment (check publish output; disable trimming of that assembly under AOT/trimming).
Add a startup smoke test that constructs the tokenizer so missing data packages fail at boot, not first use.

When it happens

Trigger: TiktokenTokenizer.CreateForModel / Create when the required data package (e.g. Microsoft.ML.Tokenizers.Data.O200kBase, .Cl100kBase, .P50kBase, .R50kBase) is not referenced, so Type.GetType(typeName) returns null and no embedded Deflate vocab stream exists.

Common situations: Upgrading Microsoft.ML.Tokenizers to 1.0+ where vocab data moved to separate NuGet packages but only the core package is installed; trimming/AOT stripping the data assembly; copying app binaries without the data DLLs.

Understand the failure class

Background: "not installed", "pip install", "required for": how missing-dependency errors surface across open-source libraries — this error's family across 34 libraries.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/2275d053482b25dc. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs:1321

                                    Normalizer? normalizer = null)
        {
            (Dictionary<string, int> SpecialTokens, Regex Regex, string VocabFile, Type? DataType, string PackageName) tiktokenConfiguration = GetTiktokenConfigurations(modelEncoding, modelName);

            if (extraSpecialTokens is not null)
            {
                foreach (var extraSpecialToken in extraSpecialTokens)
                {
                    tiktokenConfiguration.SpecialTokens.Add(extraSpecialToken.Key, extraSpecialToken.Value);
                }
            }

            if (!_tiktokenCache.TryGetValue(
                    tiktokenConfiguration.VocabFile,
                    out (Dictionary<ReadOnlyMemory<byte>, int> encoder, Dictionary<StringSpanOrdinalKey, (int Id, string Token)> vocab, Dictionary<int, ReadOnlyMemory<byte>> decoder) cache))
            {
                if (tiktokenConfiguration.DataType is null)
                {
                    throw new InvalidOperationException($"The tokenizer data file {tiktokenConfiguration.PackageName}.dll could not be loaded. Please reference the package {tiktokenConfiguration.PackageName} in your project.");
                }

                using Stream compressedStream = tiktokenConfiguration.DataType.Assembly!.GetManifestResourceStream(tiktokenConfiguration.VocabFile)!;
                using Stream deflateStream = new DeflateStream(compressedStream, CompressionMode.Decompress);

                cache = LoadTiktokenBpeAsync(deflateStream, useAsync: false).GetAwaiter().GetResult();

                _tiktokenCache.TryAdd(tiktokenConfiguration.VocabFile, cache);
            }

            return new TiktokenTokenizer(
                        cache.encoder,
                        cache.decoder,
                        cache.vocab,
                        new RegexPreTokenizer(tiktokenConfiguration.Regex, tiktokenConfiguration.SpecialTokens),
                        tiktokenConfiguration.SpecialTokens,
                        normalizer,
                        LruCache<int[]>.DefaultCacheSize);

View on GitHub (pinned to 7b76e69cf9)