What causes "Pre-tokenizer cannot be null." in dotnet/machinelearning?

Filter nulls before construction: preTokenizers.Where(p => p is not null).

dotnet/machinelearning · error · ArgumentException

Pre-tokenizer cannot be null.

Error message

Pre-tokenizer cannot be null.

What it means

CompositePreTokenizer validates that every element in the preTokenizers collection is non-null, since a null element would fail later during tokenization. Any null entry causes this ArgumentException naming the preTokenizers parameter.

Source

Thrown at src/Microsoft.ML.Tokenizers/PreTokenizer/CompositePreTokenizer.cs:50

    /// </remarks>
    public CompositePreTokenizer(IReadOnlyList<PreTokenizer> preTokenizers, IReadOnlyDictionary<string, int>? specialTokens = null)
    {
        if (preTokenizers is null)
        {
            throw new ArgumentNullException(nameof(preTokenizers));
        }

        // Limit the number of pre-tokenizers to a reasonable amount as we do a recursive calls depending on the number of pre-tokenizers
        if (preTokenizers.Count > MaxPreTokenizersCount)
        {
            throw new ArgumentException($"Too many pre-tokenizers provided. Maximum is {MaxPreTokenizersCount}.", nameof(preTokenizers));
        }

        foreach (var preTokenizer in preTokenizers)
        {
            if (preTokenizer is null)
            {
                throw new ArgumentException("Pre-tokenizer cannot be null.", nameof(preTokenizers));
            }
        }

        if (specialTokens is { Count: > 0 })
        {
            var list = new List<PreTokenizer>(specialTokens.Count + 1);

            list.Add(new RegexPreTokenizer(new Regex(string.Join("|", specialTokens.Keys.Select(s => Regex.Escape(s))), RegexOptions.Compiled), null));

            foreach (var preTokenizer in preTokenizers)
            {
                list.Add(preTokenizer);
            }

            _preTokenizers = list.AsReadOnly();
        }
        else
        {

View on GitHub (pinned to 7b76e69cf9)

Solutions

Filter nulls before construction: preTokenizers.Where(p => p is not null).
Ensure your factory/mapping returns a real PreTokenizer (e.g. a no-op) instead of null for unsupported types.
Check list initialization for default-filled slots.

Example fix

// before
var pt = new CompositePreTokenizer(steps, specialTokens); // steps contains nulls
// after
var pt = new CompositePreTokenizer(steps.Where(p => p is not null).ToList(), specialTokens);

Defensive patterns

Strategy: type-guard

Validate before calling

if (preTokenizers.Any(p => p is null))
    throw new Exception("preTokenizers collection contains null entries");

Type guard

bool AllNonNull(IEnumerable<PreTokenizer?>? list) => list is not null && list.All(p => p is not null);

Try / catch

try { var pt = new CompositePreTokenizer(preTokenizers, specialTokens); }
catch (ArgumentException ex) when (ex.Message == "Pre-tokenizer cannot be null.") {
    // filter nulls and rebuild
}

Prevention

Use Where(p => p is not null) when building the list from nullable sources
Make mapping factories return a no-op PreTokenizer instead of null for unknown types
Enable nullable reference types (PreTokenizer? vs PreTokenizer) to catch this at compile time

When it happens

Trigger: new CompositePreTokenizer(new PreTokenizer?[] { new WhitespacePreTokenizer(), null }, null) — a null element anywhere in the collection, often from LINQ projections that can yield null or conditional construction returning null.

Common situations: Mapping a tokenizer.json pre_tokenizer list to .NET types where an unsupported type maps to null; arrays initialized with default entries; factory methods returning null for unknown steps.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/6537e836654557d3. Report an issue: GitHub.