{"record":{"id":"0f4f29a214fbacb5","repo":"dotnet/machinelearning","slug":"the-maximum-number-of-tokens-must-be-greater-than","errorCode":null,"errorMessage":"The maximum number of tokens must be greater than zero.","messagePattern":"The maximum number of tokens must be greater than zero\\.","errorType":"validation","errorClass":"ArgumentOutOfRangeException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs","lineNumber":494,"sourceCode":"                tokens.Add(new EncodedToken(EndOfSentenceId, EndOfSentenceToken, new Range(charsConsumed, charsConsumed)));\n            }\n\n            return new EncodeResults<EncodedToken> { Tokens = tokens, NormalizedText = normalizedText, CharsConsumed = charsConsumed };\n        }\n\n        /// <summary>\n        /// Encodes input text to token Ids.\n        /// </summary>\n        /// <param name=\"text\">The text to encode.</param>\n        /// <param name=\"textSpan\">The span of the text to encode which will be used if the <paramref name=\"text\"/> is <see langword=\"null\"/>.</param>\n        /// <param name=\"settings\">The settings used to encode the text.</param>\n        /// <returns>The encoded results containing the list of encoded Ids.</returns>\n        protected override EncodeResults<int> EncodeToIds(string? text, ReadOnlySpan<char> textSpan, EncodeSettings settings)\n        {\n            int maxTokenCount = settings.MaxTokenCount;\n            if (maxTokenCount <= 0)\n            {\n                throw new ArgumentOutOfRangeException(nameof(settings.MaxTokenCount), \"The maximum number of tokens must be greater than zero.\");\n            }\n\n            if (string.IsNullOrEmpty(text) && textSpan.IsEmpty)\n            {\n                return new EncodeResults<int> { Tokens = [], NormalizedText = null, CharsConsumed = 0 };\n            }\n\n            IEnumerable<(int Offset, int Length)>? splits = InitializeForEncoding(\n                                                                text,\n                                                                textSpan,\n                                                                settings.ConsiderPreTokenization,\n                                                                settings.ConsiderNormalization,\n                                                                _normalizer,\n                                                                _preTokenizer,\n                                                                out string? normalizedText,\n                                                                out ReadOnlySpan<char> textSpanToEncode,\n                                                                out _);\n","sourceCodeStart":476,"sourceCodeEnd":512,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs#L476-L512","documentation":"EncodeToIds enforces that settings.MaxTokenCount is strictly greater than zero and throws ArgumentOutOfRangeException otherwise. The parameter controls the token-budget cap during encoding; a zero or negative value has no valid meaning and indicates a caller bug.","triggerScenarios":"Calling tokenizer.EncodeToIds(text, maxTokenCount: 0), EncodeToIds(text, settings: new EncodeSettings { MaxTokenCount = -1 }), or default-initialized settings structs where MaxTokenCount was never set to a positive value.","commonSituations":"Reading MaxTokenCount from config that yields 0 (unset int / empty string parsed to 0); passing a user-supplied 'max length' of 0 meaning 'unlimited' instead of using int.MaxValue or no limit; arithmetic like totalBudget - alreadyUsed going negative.","solutions":["Pass a positive maxTokenCount (e.g. the model's context length).","If you need no limit, pass int.MaxValue instead of 0.","Clamp computed budgets: Math.Max(1, budget - consumed).","Validate the config value > 0 before constructing EncodeSettings."],"exampleFix":"// before\nint max = config.MaxTokens; // 0 when unset\nvar ids = tokenizer.EncodeToIds(text, maxTokenCount: max);\n// after\nint max = config.MaxTokens > 0 ? config.MaxTokens : int.MaxValue;\nvar ids = tokenizer.EncodeToIds(text, maxTokenCount: max);","handlingStrategy":"validation","validationCode":"if (maxTokenCount <= 0) maxTokenCount = modelContextLength; // or int.MaxValue for no cap\nvar ids = tokenizer.EncodeToIds(text, maxTokenCount: maxTokenCount);","typeGuard":"static int PositiveTokenLimit(int n) => n > 0 ? n : int.MaxValue;","tryCatchPattern":"try { var ids = tokenizer.EncodeToIds(text, settings); }\ncatch (ArgumentOutOfRangeException ex) when (ex.ParamName == \"settings.MaxTokenCount\") { settings.MaxTokenCount = int.MaxValue; var ids = tokenizer.EncodeToIds(text, settings); }","preventionTips":["Treat 0/negative max token counts as 'unlimited' in your own config layer and translate to int.MaxValue.","Clamp remaining-budget arithmetic with Math.Max(1, remaining).","Initialize EncodeSettings.MaxTokenCount explicitly; never rely on defaults."],"tags":["argument-out-of-range","encoding","bpe","tokenizer"],"backgroundTag":"argument-out-of-range","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}