{"record":{"id":"45cd80696efc5031","repo":"dotnet/machinelearning","slug":"the-maximum-number-of-tokens-must-be-greater-than-45cd80","errorCode":null,"errorMessage":"The maximum number of tokens must be greater than zero.","messagePattern":"The maximum number of tokens must be greater than zero\\.","errorType":"exception","errorClass":"ArgumentOutOfRangeException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs","lineNumber":367,"sourceCode":"                                encodedTokens[i].Id,\n                                encodedTokens[i].TokenLength == 0 ? string.Empty : text.Slice(encodedTokens[i].TokenIndex, encodedTokens[i].TokenLength).ToString(),\n                                new Range(encodedTokens[i].TokenIndex + offset, encodedTokens[i].TokenIndex + offset + encodedTokens[i].TokenLength)));\n            }\n        }\n\n        /// <summary>\n        /// Encodes input text to token Ids.\n        /// </summary>\n        /// <param name=\"text\">The text to encode.</param>\n        /// <param name=\"textSpan\">The span of the text to encode which will be used if the <paramref name=\"text\"/> is <see langword=\"null\"/>.</param>\n        /// <param name=\"settings\">The settings used to encode the text.</param>\n        /// <returns>The encoded results containing the list of encoded Ids.</returns>\n        protected override EncodeResults<int> EncodeToIds(string? text, ReadOnlySpan<char> textSpan, EncodeSettings settings)\n        {\n            int maxTokenCount = settings.MaxTokenCount;\n            if (maxTokenCount <= 0)\n            {\n                throw new ArgumentOutOfRangeException(nameof(settings.MaxTokenCount), \"The maximum number of tokens must be greater than zero.\");\n            }\n\n            if (string.IsNullOrEmpty(text) && textSpan.IsEmpty)\n            {\n                return new EncodeResults<int> { NormalizedText = null, Tokens = [], CharsConsumed = 0 };\n            }\n\n            IEnumerable<(int Offset, int Length)>? splits = InitializeForEncoding(\n                                                                text,\n                                                                textSpan,\n                                                                settings.ConsiderPreTokenization,\n                                                                settings.ConsiderNormalization,\n                                                                _normalizer,\n                                                                _preTokenizer,\n                                                                out string? normalizedText,\n                                                                out ReadOnlySpan<char> textSpanToEncode,\n                                                                out int charsConsumed);\n","sourceCodeStart":349,"sourceCodeEnd":385,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs#L349-L385","documentation":"Argument guard in TiktokenTokenizer.EncodeToIds (and its overloads): the maximum token count allowed for the encoded output must be greater than zero. Passing 0 or a negative value would make the encoding budget meaningless (no tokens could ever be produced), so the method validates the limit up front and throws before any text is encoded.","triggerScenarios":"Calling EncodeToIds (or the public EncodeToIds overloads) with an EncodeSettings whose MaxTokenCount is 0 or negative, e.g. 'new EncodeSettings { MaxTokenCount = 0 }'.","commonSituations":"Computing MaxTokenCount dynamically (e.g. promptWindowSize - promptTokens) so it underflows to 0 or negative; copying MaxTokenCount from an uninitialized int field; passing default structs misused as 'unlimited'.","solutions":["Pass MaxTokenCount = int.MaxValue (the default) when you want no limit instead of 0.","Clamp computed budgets to at least 1 before calling: Math.Max(1, remaining).","Ensure the settings object actually sets MaxTokenCount only when a valid positive limit is intended.","Guard inputs before encoding."],"exampleFix":"// before\nvar r = tok.EncodeToIds(text, new EncodeSettings { MaxTokenCount = 0 });\n// after\nvar r = tok.EncodeToIds(text, new EncodeSettings { MaxTokenCount = Math.Max(1, remaining) });","handlingStrategy":"validation","validationCode":"static int SafeMaxTokenCount(int requested) =>\n    requested <= 0 ? int.MaxValue : requested;\n// usage: settings.MaxTokenCount = SafeMaxTokenCount(computedBudget);","typeGuard":"static bool IsValidMaxTokenCount(int? v) => v is null || v > 0;","tryCatchPattern":"try { var r = tok.EncodeToIds(text, settings); }\ncatch (ArgumentOutOfRangeException) { settings.MaxTokenCount = int.MaxValue; var r = tok.EncodeToIds(text, settings); }","preventionTips":["Treat 0/negative budgets as 'unlimited' by mapping to int.MaxValue at your boundary.","Clamp dynamically computed budgets (contextWindow - promptTokens) with Math.Max(1, x).","Never leave MaxTokenCount as a default 0 in settings objects you build manually."],"tags":["tokenizer","argument-out-of-range","max-token-count","encoding"],"backgroundTag":"argument-out-of-range","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}