{"record":{"id":"cdf140b768652180","repo":"dotnet/machinelearning","slug":"the-maximum-number-of-tokens-must-be-greater-than-cdf140","errorCode":null,"errorMessage":"The maximum number of tokens must be greater than zero.","messagePattern":"The maximum number of tokens must be greater than zero\\.","errorType":"exception","errorClass":"ArgumentOutOfRangeException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/WordPieceTokenizer.cs","lineNumber":394,"sourceCode":"            if (arrayPool is not null)\n            {\n                ArrayPool<char>.Shared.Return(arrayPool);\n            }\n        }\n\n        /// <summary>\n        /// Encodes input text to token Ids.\n        /// </summary>\n        /// <param name=\"text\">The text to encode.</param>\n        /// <param name=\"textSpan\">The span of the text to encode which will be used if the <paramref name=\"text\"/> is <see langword=\"null\"/>.</param>\n        /// <param name=\"settings\">The settings used to encode the text.</param>\n        /// <returns>The encoded results containing the list of encoded Ids.</returns>\n        protected override EncodeResults<int> EncodeToIds(string? text, ReadOnlySpan<char> textSpan, EncodeSettings settings)\n        {\n            int maxTokenCount = settings.MaxTokenCount;\n            if (maxTokenCount <= 0)\n            {\n                throw new ArgumentOutOfRangeException(nameof(settings.MaxTokenCount), \"The maximum number of tokens must be greater than zero.\");\n            }\n\n            if (string.IsNullOrEmpty(text) && textSpan.IsEmpty)\n            {\n                return new EncodeResults<int> { NormalizedText = null, Tokens = [], CharsConsumed = 0 };\n            }\n\n            IEnumerable<(int Offset, int Length)>? splits = InitializeForEncoding(\n                                                                text,\n                                                                textSpan,\n                                                                settings.ConsiderPreTokenization,\n                                                                settings.ConsiderNormalization,\n                                                                _normalizer,\n                                                                _preTokenizer,\n                                                                out string? normalizedText,\n                                                                out ReadOnlySpan<char> textSpanToEncode,\n                                                                out int charsConsumed);\n","sourceCodeStart":376,"sourceCodeEnd":412,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/WordPieceTokenizer.cs#L376-L412","documentation":"WordPieceTokenizer.EncodeToIds validates settings.MaxTokenCount and throws ArgumentOutOfRangeException when it is <= 0, since a non-positive cap on output tokens has no meaning. This mirrors validation in all encoder paths of the tokenizer.","triggerScenarios":"Calling tokenizer.EncodeToIds(text, maxTokenCount: 0) or EncodeToIds(text, settings) with settings.MaxTokenCount = 0 or negative — commonly the default int value when a MaxTokenCount variable was never assigned.","commonSituations":"Passing 0 intending 'no limit' (must instead use int.MaxValue); reading maxTokenCount from config where it defaulted to 0; uninitialized field in a wrapper class.","solutions":["Pass a positive maxTokenCount, or int.MaxValue for unlimited encoding","Fix the configuration/source so the limit defaults to a positive value","Guard the call site: if (limit <= 0) limit = int.MaxValue;"],"exampleFix":"// before\nvar ids = tokenizer.EncodeToIds(text, 0);\n// after\nvar ids = tokenizer.EncodeToIds(text, int.MaxValue);","handlingStrategy":"validation","validationCode":"if (maxTokenCount <= 0) maxTokenCount = int.MaxValue; // unlimited","typeGuard":"bool IsValidMaxTokenCount(int n) => n > 0;","tryCatchPattern":"try { var res = tokenizer.EncodeToIds(text, maxTokenCount); } catch (ArgumentOutOfRangeException ex) when (ex.ParamName == \"settings.MaxTokenCount\") { var res2 = tokenizer.EncodeToIds(text, int.MaxValue); }","preventionTips":["Use int.MaxValue, never 0, to mean 'no limit'","Validate configured max-length values at startup","Wrap encode calls in a helper that normalizes the limit"],"tags":["tokenizers","argument-out-of-range","encode"],"backgroundTag":"argument-out-of-range","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}