{"record":{"id":"2275d053482b25dc","repo":"dotnet/machinelearning","slug":"the-tokenizer-data-file-tiktokenconfiguration-pac","errorCode":null,"errorMessage":"The tokenizer data file {tiktokenConfiguration.PackageName}.dll could not be loaded. Please reference the package {tiktokenConfiguration.PackageName} in your project.","messagePattern":"The tokenizer data file (.+?)\\.dll could not be loaded\\. Please reference the package (.+?) in your project\\.","errorType":"exception","errorClass":"InvalidOperationException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs","lineNumber":1321,"sourceCode":"                                    Normalizer? normalizer = null)\n        {\n            (Dictionary<string, int> SpecialTokens, Regex Regex, string VocabFile, Type? DataType, string PackageName) tiktokenConfiguration = GetTiktokenConfigurations(modelEncoding, modelName);\n\n            if (extraSpecialTokens is not null)\n            {\n                foreach (var extraSpecialToken in extraSpecialTokens)\n                {\n                    tiktokenConfiguration.SpecialTokens.Add(extraSpecialToken.Key, extraSpecialToken.Value);\n                }\n            }\n\n            if (!_tiktokenCache.TryGetValue(\n                    tiktokenConfiguration.VocabFile,\n                    out (Dictionary<ReadOnlyMemory<byte>, int> encoder, Dictionary<StringSpanOrdinalKey, (int Id, string Token)> vocab, Dictionary<int, ReadOnlyMemory<byte>> decoder) cache))\n            {\n                if (tiktokenConfiguration.DataType is null)\n                {\n                    throw new InvalidOperationException($\"The tokenizer data file {tiktokenConfiguration.PackageName}.dll could not be loaded. Please reference the package {tiktokenConfiguration.PackageName} in your project.\");\n                }\n\n                using Stream compressedStream = tiktokenConfiguration.DataType.Assembly!.GetManifestResourceStream(tiktokenConfiguration.VocabFile)!;\n                using Stream deflateStream = new DeflateStream(compressedStream, CompressionMode.Decompress);\n\n                cache = LoadTiktokenBpeAsync(deflateStream, useAsync: false).GetAwaiter().GetResult();\n\n                _tiktokenCache.TryAdd(tiktokenConfiguration.VocabFile, cache);\n            }\n\n            return new TiktokenTokenizer(\n                        cache.encoder,\n                        cache.decoder,\n                        cache.vocab,\n                        new RegexPreTokenizer(tiktokenConfiguration.Regex, tiktokenConfiguration.SpecialTokens),\n                        tiktokenConfiguration.SpecialTokens,\n                        normalizer,\n                        LruCache<int[]>.DefaultCacheSize);","sourceCodeStart":1303,"sourceCodeEnd":1339,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs#L1303-L1339","documentation":"TiktokenTokenizer resolves vocab data from an embedded resource inside a data-package assembly (Microsoft.ML.Tokenizers.Data.*). If the cached entry is missing and tiktokenConfiguration.DataType is null — meaning the data-package type/assembly could not be located — an InvalidOperationException telling you to reference the data package is thrown.","triggerScenarios":"TiktokenTokenizer.CreateForModel / Create when the required data package (e.g. Microsoft.ML.Tokenizers.Data.O200kBase, .Cl100kBase, .P50kBase, .R50kBase) is not referenced, so Type.GetType(typeName) returns null and no embedded Deflate vocab stream exists.","commonSituations":"Upgrading Microsoft.ML.Tokenizers to 1.0+ where vocab data moved to separate NuGet packages but only the core package is installed; trimming/AOT stripping the data assembly; copying app binaries without the data DLLs.","solutions":["Add the matching NuGet package: Microsoft.ML.Tokenizers.Data.O200kBase (GPT-4o/o200k), .Cl100kBase (GPT-4/3.5), or .P50kBase/.R50kBase for older models.","Run 'dotnet add package Microsoft.ML.Tokenizers.Data.O200kBase' (or the relevant one) and rebuild so the data DLL is deployed with the app.","If self-hosting, ensure the data package assembly is copied to the output and loadable at runtime (check trimming/AOT settings).","Alternatively supply your own vocab stream via TiktokenTokenizer.CreateAsync(stream, specialTokens) instead of relying on built-in data."],"exampleFix":"// before\ndotnet add package Microsoft.ML.Tokenizers\nvar tok = TiktokenTokenizer.CreateForModel(\"gpt-4o\"); // InvalidOperationException: ...Data.O200kBase.dll could not be loaded\n// after\ndotnet add package Microsoft.ML.Tokenizers\ndotnet add package Microsoft.ML.Tokenizers.Data.O200kBase\nvar tok = TiktokenTokenizer.CreateForModel(\"gpt-4o\");","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try { var tok = TiktokenTokenizer.CreateForModel(\"gpt-4o\"); }\ncatch (InvalidOperationException ex) when (ex.Message.Contains(\"could not be loaded\"))\n{\n    throw new InvalidOperationException(\n        \"Missing tokenizer data package. Add e.g. Microsoft.ML.Tokenizers.Data.O200kBase to your project.\", ex);\n}","preventionTips":["Reference the required data package (Data.O200kBase / Data.Cl100kBase / Data.P50kBase / Data.R50kBase) whenever you upgrade to Microsoft.ML.Tokenizers 1.0+.","Verify the data DLL ships with your deployment (check publish output; disable trimming of that assembly under AOT/trimming).","Add a startup smoke test that constructs the tokenizer so missing data packages fail at boot, not first use."],"tags":["tokenizer","missing-dependency","nuget-package","tiktoken-data"],"backgroundTag":"missing-dependency","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}