{"record":{"id":"70bfd6bf89a371d7","repo":"dotnet/machinelearning","slug":"the-max-token-count-must-be-greater-than-0-70bfd6","errorCode":null,"errorMessage":"The max token count must be greater than 0.","messagePattern":"The max token count must be greater than 0\\.","errorType":"exception","errorClass":"ArgumentOutOfRangeException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs","lineNumber":661,"sourceCode":"        /// If <paramRef name=\"fromEnd\" /> is <see langword=\"true\"/>, it represents the index of the first character to be included. In cases where no tokens fit, the result will be the text length; conversely,\n        /// if all tokens fit, the result will be zero.\n        /// </returns>\n        protected override int GetIndexByTokenCount(string? text, ReadOnlySpan<char> textSpan, EncodeSettings settings, bool fromEnd, out string? normalizedText, out int tokenCount)\n        {\n            if (fromEnd)\n            {\n                return LastIndexOf(text, textSpan, settings.MaxTokenCount, settings.ConsiderNormalization, settings.ConsiderNormalization, out normalizedText, out tokenCount);\n            }\n\n            tokenCount = CountTokens(text, textSpan, settings.ConsiderPreTokenization, settings.ConsiderNormalization, out normalizedText, out int charsConsumed, settings.MaxTokenCount);\n            return charsConsumed;\n        }\n\n        private int LastIndexOf(string? text, ReadOnlySpan<char> textSpan, int maxTokenCount, bool considerPreTokenization, bool considerNormalization, out string? normalizedText, out int tokenCount)\n        {\n            if (maxTokenCount <= 0)\n            {\n                throw new ArgumentOutOfRangeException(nameof(maxTokenCount), \"The max token count must be greater than 0.\");\n            }\n\n            if (string.IsNullOrEmpty(text) && textSpan.IsEmpty)\n            {\n                normalizedText = null;\n                tokenCount = 0;\n                return 0;\n            }\n\n            IEnumerable<(int Offset, int Length)>? splits = InitializeForEncoding(\n                                                                text,\n                                                                textSpan,\n                                                                considerPreTokenization,\n                                                                considerNormalization,\n                                                                _normalizer,\n                                                                _preTokenizer,\n                                                                out normalizedText,\n                                                                out ReadOnlySpan<char> textSpanToEncode,","sourceCodeStart":643,"sourceCodeEnd":679,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs#L643-L679","documentation":"The private LastIndexOf helper (used by GetIndexByTokenCount) throws ArgumentOutOfRangeException with the message 'The max token count must be greater than 0.' when maxTokenCount <= 0. Note the wording differs slightly from EncodeToIds/CountTokens but the rule is identical.","triggerScenarios":"Calling GetIndexByTokenCount with a maxTokenCount argument of 0 or a negative value; GetIndexByTokenCount forwards the caller's EncodeSettings.MaxTokenCount into LastIndexOf.","commonSituations":"Search-back logic computing a decreasing token budget that reaches 0; uninitialized settings; off-by-one in chunking loops.","solutions":["Ensure the value passed to GetIndexByTokenCount/maxTokenCount is >= 1.","Clamp: var mtc = Math.Max(1, requested);","Fix chunking loops so they terminate before the budget hits zero.","Unify validation and messages across the tokenizer API by validating at your own boundary first."],"exampleFix":"// before\nvar idx = tok.GetIndexByTokenCount(text, 0, out int count);\n// after\nvar idx = tok.GetIndexByTokenCount(text, Math.Max(1, budget), out int count);","handlingStrategy":"validation","validationCode":"if (maxTokenCount <= 0)\n    throw new ArgumentException(\"maxTokenCount must be >= 1\", nameof(maxTokenCount));\nvar idx = tok.GetIndexByTokenCount(text, maxTokenCount, out int tokenCount);","typeGuard":"static bool IsValidMaxTokenCount(int v) => v > 0;","tryCatchPattern":"try { var idx = tok.GetIndexByTokenCount(text, mtc, out int count); }\ncatch (ArgumentOutOfRangeException)\n{ mtc = Math.Max(1, mtc); var idx = tok.GetIndexByTokenCount(text, mtc, out int count); }","preventionTips":["In chunking/search-back loops, clamp the remaining budget to >= 1 before each call.","Initialize budgets from positive constants, never uninitialized fields.","Add a debug assertion maxTokenCount > 0 in your tokenizer helper layer."],"tags":["tokenizer","argument-out-of-range","max-token-count","get-index-by-token-count"],"backgroundTag":"argument-out-of-range","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}