{"record":{"id":"6537e836654557d3","repo":"dotnet/machinelearning","slug":"pre-tokenizer-cannot-be-null","errorCode":null,"errorMessage":"Pre-tokenizer cannot be null.","messagePattern":"Pre-tokenizer cannot be null\\.","errorType":"validation","errorClass":"ArgumentException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/PreTokenizer/CompositePreTokenizer.cs","lineNumber":50,"sourceCode":"    /// </remarks>\n    public CompositePreTokenizer(IReadOnlyList<PreTokenizer> preTokenizers, IReadOnlyDictionary<string, int>? specialTokens = null)\n    {\n        if (preTokenizers is null)\n        {\n            throw new ArgumentNullException(nameof(preTokenizers));\n        }\n\n        // Limit the number of pre-tokenizers to a reasonable amount as we do a recursive calls depending on the number of pre-tokenizers\n        if (preTokenizers.Count > MaxPreTokenizersCount)\n        {\n            throw new ArgumentException($\"Too many pre-tokenizers provided. Maximum is {MaxPreTokenizersCount}.\", nameof(preTokenizers));\n        }\n\n        foreach (var preTokenizer in preTokenizers)\n        {\n            if (preTokenizer is null)\n            {\n                throw new ArgumentException(\"Pre-tokenizer cannot be null.\", nameof(preTokenizers));\n            }\n        }\n\n        if (specialTokens is { Count: > 0 })\n        {\n            var list = new List<PreTokenizer>(specialTokens.Count + 1);\n\n            list.Add(new RegexPreTokenizer(new Regex(string.Join(\"|\", specialTokens.Keys.Select(s => Regex.Escape(s))), RegexOptions.Compiled), null));\n\n            foreach (var preTokenizer in preTokenizers)\n            {\n                list.Add(preTokenizer);\n            }\n\n            _preTokenizers = list.AsReadOnly();\n        }\n        else\n        {","sourceCodeStart":32,"sourceCodeEnd":68,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/PreTokenizer/CompositePreTokenizer.cs#L32-L68","documentation":"CompositePreTokenizer validates that every element in the preTokenizers collection is non-null, since a null element would fail later during tokenization. Any null entry causes this ArgumentException naming the preTokenizers parameter.","triggerScenarios":"new CompositePreTokenizer(new PreTokenizer?[] { new WhitespacePreTokenizer(), null }, null) — a null element anywhere in the collection, often from LINQ projections that can yield null or conditional construction returning null.","commonSituations":"Mapping a tokenizer.json pre_tokenizer list to .NET types where an unsupported type maps to null; arrays initialized with default entries; factory methods returning null for unknown steps.","solutions":["Filter nulls before construction: preTokenizers.Where(p => p is not null).","Ensure your factory/mapping returns a real PreTokenizer (e.g. a no-op) instead of null for unsupported types.","Check list initialization for default-filled slots."],"exampleFix":"// before\nvar pt = new CompositePreTokenizer(steps, specialTokens); // steps contains nulls\n// after\nvar pt = new CompositePreTokenizer(steps.Where(p => p is not null).ToList(), specialTokens);","handlingStrategy":"type-guard","validationCode":"if (preTokenizers.Any(p => p is null))\n    throw new Exception(\"preTokenizers collection contains null entries\");","typeGuard":"bool AllNonNull(IEnumerable<PreTokenizer?>? list) => list is not null && list.All(p => p is not null);","tryCatchPattern":"try { var pt = new CompositePreTokenizer(preTokenizers, specialTokens); }\ncatch (ArgumentException ex) when (ex.Message == \"Pre-tokenizer cannot be null.\") {\n    // filter nulls and rebuild\n}","preventionTips":["Use Where(p => p is not null) when building the list from nullable sources","Make mapping factories return a no-op PreTokenizer instead of null for unknown types","Enable nullable reference types (PreTokenizer? vs PreTokenizer) to catch this at compile time"],"tags":["tokenizer","pre-tokenizer","null-argument"],"backgroundTag":"null-argument","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}