dotnet/machinelearning · error · ArgumentException

Too many pre-tokenizers provided. Maximum is {MaxPreTokenize

Error message

Too many pre-tokenizers provided. Maximum is {MaxPreTokenizersCount}.

What it means

CompositePreTokenizer aggregates pre-tokenizers and recursively calls into them, so the count is capped at MaxPreTokenizersCount to avoid deep recursion/exponential behavior. Passing a collection with more entries than the cap throws this ArgumentException naming the preTokenizers parameter.

Source

Thrown at src/Microsoft.ML.Tokenizers/PreTokenizer/CompositePreTokenizer.cs:43

    /// </summary>
    /// <param name="preTokenizers">The list of pre-tokenizers to apply.</param>
    /// <param name="specialTokens">The special tokens to apply.</param>
    /// <exception cref="ArgumentNullException">Thrown when <paramref name="preTokenizers"/> is null.</exception>
    /// <exception cref="ArgumentException">Thrown when <paramref name="preTokenizers"/> contains null elements.</exception>
    /// <remarks>
    /// The <see cref="CompositePreTokenizer"/> can accept a list of pre-tokenizers with a maximum of 10 items.
    /// </remarks>
    public CompositePreTokenizer(IReadOnlyList<PreTokenizer> preTokenizers, IReadOnlyDictionary<string, int>? specialTokens = null)
    {
        if (preTokenizers is null)
        {
            throw new ArgumentNullException(nameof(preTokenizers));
        }

        // Limit the number of pre-tokenizers to a reasonable amount as we do a recursive calls depending on the number of pre-tokenizers
        if (preTokenizers.Count > MaxPreTokenizersCount)
        {
            throw new ArgumentException($"Too many pre-tokenizers provided. Maximum is {MaxPreTokenizersCount}.", nameof(preTokenizers));
        }

        foreach (var preTokenizer in preTokenizers)
        {
            if (preTokenizer is null)
            {
                throw new ArgumentException("Pre-tokenizer cannot be null.", nameof(preTokenizers));
            }
        }

        if (specialTokens is { Count: > 0 })
        {
            var list = new List<PreTokenizer>(specialTokens.Count + 1);

            list.Add(new RegexPreTokenizer(new Regex(string.Join("|", specialTokens.Keys.Select(s => Regex.Escape(s))), RegexOptions.Compiled), null));

            foreach (var preTokenizer in preTokenizers)
            {

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Reduce the number of pre-tokenizers — merge adjacent Whitespace/ByteLevel style steps where equivalent.
2. Split processing into multiple sequential CompositePreTokenizer stages in your own pipeline.
3. Simplify the tokenizer.json pre_tokenizer sequence with the HuggingFace tokenizers library before loading.

Example fix

// before: 200 steps in one composite
var pt = new CompositePreTokenizer(allSteps, specialTokens);
// after: chunk into batches under the cap
foreach (var chunk in allSteps.Chunk(MaxPreTokenizersCount))
    text = new CompositePreTokenizer(chunk, null).PreTokenize(text);

if (preTokenizers.Count > CompositePreTokenizer.MaxPreTokenizersCount)
    throw new Exception($"Too many pre-tokenizers ({preTokenizers.Count}); merge or split them.");

bool WithinPreTokenizerLimit(IReadOnlyList<PreTokenizer> list) => list.Count <= CompositePreTokenizer.MaxPreTokenizersCount;

try { var pt = new CompositePreTokenizer(preTokenizers, specialTokens); }
catch (ArgumentException ex) when (ex.Message.Contains("Too many pre-tokenizers")) {
    // chunk the list or merge equivalent steps
}

When it happens

Trigger: new CompositePreTokenizer(list, specialTokens) where list.Count > MaxPreTokenizersCount (value in the library constant) — e.g. building a pre-tokenizer chain programmatically from a tokenizer.json with many combined pre-tokenizers.

Common situations: Auto-converting very complex tokenizer.json pre_tokenizer sequences into a flat list; concatenating multiple pipelines into one composite; generated configs that never merge steps.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• The tokenizer.json pre_tokenizer type '{type ?? "<missing>"}
• Blob for normalization rule is broken.
• Pre-tokenizer cannot be null.
• failed to insert key: negative value
• Exception of type 'System.ArgumentException' was thrown.

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/09ca8ce40b6409f8. Report an issue: GitHub.