{"record":{"id":"09ca8ce40b6409f8","repo":"dotnet/machinelearning","slug":"too-many-pre-tokenizers-provided-maximum-is-maxp","errorCode":null,"errorMessage":"Too many pre-tokenizers provided. Maximum is {MaxPreTokenizersCount}.","messagePattern":"Too many pre-tokenizers provided\\. Maximum is (.+?)\\.","errorType":"validation","errorClass":"ArgumentException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/PreTokenizer/CompositePreTokenizer.cs","lineNumber":43,"sourceCode":"    /// </summary>\n    /// <param name=\"preTokenizers\">The list of pre-tokenizers to apply.</param>\n    /// <param name=\"specialTokens\">The special tokens to apply.</param>\n    /// <exception cref=\"ArgumentNullException\">Thrown when <paramref name=\"preTokenizers\"/> is null.</exception>\n    /// <exception cref=\"ArgumentException\">Thrown when <paramref name=\"preTokenizers\"/> contains null elements.</exception>\n    /// <remarks>\n    /// The <see cref=\"CompositePreTokenizer\"/> can accept a list of pre-tokenizers with a maximum of 10 items.\n    /// </remarks>\n    public CompositePreTokenizer(IReadOnlyList<PreTokenizer> preTokenizers, IReadOnlyDictionary<string, int>? specialTokens = null)\n    {\n        if (preTokenizers is null)\n        {\n            throw new ArgumentNullException(nameof(preTokenizers));\n        }\n\n        // Limit the number of pre-tokenizers to a reasonable amount as we do a recursive calls depending on the number of pre-tokenizers\n        if (preTokenizers.Count > MaxPreTokenizersCount)\n        {\n            throw new ArgumentException($\"Too many pre-tokenizers provided. Maximum is {MaxPreTokenizersCount}.\", nameof(preTokenizers));\n        }\n\n        foreach (var preTokenizer in preTokenizers)\n        {\n            if (preTokenizer is null)\n            {\n                throw new ArgumentException(\"Pre-tokenizer cannot be null.\", nameof(preTokenizers));\n            }\n        }\n\n        if (specialTokens is { Count: > 0 })\n        {\n            var list = new List<PreTokenizer>(specialTokens.Count + 1);\n\n            list.Add(new RegexPreTokenizer(new Regex(string.Join(\"|\", specialTokens.Keys.Select(s => Regex.Escape(s))), RegexOptions.Compiled), null));\n\n            foreach (var preTokenizer in preTokenizers)\n            {","sourceCodeStart":25,"sourceCodeEnd":61,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/PreTokenizer/CompositePreTokenizer.cs#L25-L61","documentation":"CompositePreTokenizer aggregates pre-tokenizers and recursively calls into them, so the count is capped at MaxPreTokenizersCount to avoid deep recursion/exponential behavior. Passing a collection with more entries than the cap throws this ArgumentException naming the preTokenizers parameter.","triggerScenarios":"new CompositePreTokenizer(list, specialTokens) where list.Count > MaxPreTokenizersCount (value in the library constant) — e.g. building a pre-tokenizer chain programmatically from a tokenizer.json with many combined pre-tokenizers.","commonSituations":"Auto-converting very complex tokenizer.json pre_tokenizer sequences into a flat list; concatenating multiple pipelines into one composite; generated configs that never merge steps.","solutions":["Reduce the number of pre-tokenizers — merge adjacent Whitespace/ByteLevel style steps where equivalent.","Split processing into multiple sequential CompositePreTokenizer stages in your own pipeline.","Simplify the tokenizer.json pre_tokenizer sequence with the HuggingFace tokenizers library before loading."],"exampleFix":"// before: 200 steps in one composite\nvar pt = new CompositePreTokenizer(allSteps, specialTokens);\n// after: chunk into batches under the cap\nforeach (var chunk in allSteps.Chunk(MaxPreTokenizersCount))\n    text = new CompositePreTokenizer(chunk, null).PreTokenize(text);","handlingStrategy":"validation","validationCode":"if (preTokenizers.Count > CompositePreTokenizer.MaxPreTokenizersCount)\n    throw new Exception($\"Too many pre-tokenizers ({preTokenizers.Count}); merge or split them.\");","typeGuard":"bool WithinPreTokenizerLimit(IReadOnlyList<PreTokenizer> list) => list.Count <= CompositePreTokenizer.MaxPreTokenizersCount;","tryCatchPattern":"try { var pt = new CompositePreTokenizer(preTokenizers, specialTokens); }\ncatch (ArgumentException ex) when (ex.Message.Contains(\"Too many pre-tokenizers\")) {\n    // chunk the list or merge equivalent steps\n}","preventionTips":["Merge redundant pre-tokenizer steps before composing","Keep tokenizer.json pre_tokenizer chains short","Check the count against MaxPreTokenizersCount in config pipelines"],"tags":["tokenizer","pre-tokenizer","argument","limit"],"backgroundTag":"value-out-of-range","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}