{"record":{"id":"d8072de5795733ad","repo":"dotnet/machinelearning","slug":"the-maximum-number-of-tokens-must-be-greater-than-d8072d","errorCode":null,"errorMessage":"The maximum number of tokens must be greater than zero.","messagePattern":"The maximum number of tokens must be greater than zero\\.","errorType":"exception","errorClass":"ArgumentOutOfRangeException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs","lineNumber":582,"sourceCode":"        {\n            return EncodeToIds(null, text, addPrefixSpace, addBeginningOfSentence, addEndOfSentence, considerPreTokenization, considerNormalization, out normalizedText, out charsConsumed, maxTokenCount);\n        }\n\n        private IReadOnlyList<int> EncodeToIds(\n                                    string? text,\n                                    scoped ReadOnlySpan<char> textSpan,\n                                    bool addPrefixSpace,\n                                    bool addBeginningOfSentence,\n                                    bool addEndOfSentence,\n                                    bool considerPreTokenization,\n                                    bool considerNormalization,\n                                    out string? normalizedText,\n                                    out int charsConsumed,\n                                    int maxTokenCount = int.MaxValue)\n        {\n            if (maxTokenCount <= 0)\n            {\n                throw new ArgumentOutOfRangeException(nameof(maxTokenCount), \"The maximum number of tokens must be greater than zero.\");\n            }\n\n            if (string.IsNullOrEmpty(text) && textSpan.IsEmpty)\n            {\n                charsConsumed = 0;\n                normalizedText = null;\n                return [];\n            }\n\n            char[]? mutatedInputText = null;\n\n            try\n            {\n                Span<char> mutatedInputSpan = stackalloc char[BufferLength];\n                scoped ReadOnlySpan<char> textSpanToEncode;\n                IEnumerable<(int Offset, int Length)>? splits;\n                if (addPrefixSpace)\n                {","sourceCodeStart":564,"sourceCodeEnd":600,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs#L564-L600","documentation":"EncodeToIds validates that maxTokenCount is a positive integer before encoding; zero or negative values are meaningless caps and throw ArgumentOutOfRangeException naming the parameter. This is a standard guard on the public EncodeToIds overloads accepting string or ReadOnlySpan<char>.","triggerScenarios":"Calling tokenizer.EncodeToIds(text or span, ..., maxTokenCount) with maxTokenCount = 0 or negative, often a default-initialized int or a computed limit like text.Length when the text is empty.","commonSituations":"A caller-supplied 'budget' variable that was never set (defaults to 0), or off-by-one arithmetic producing a non-positive cap.","solutions":["Pass maxTokenCount > 0, or omit it to use the int.MaxValue default.","Clamp computed limits: Math.Max(1, computedLimit) before calling.","Treat 0 as 'no limit' at the call site by passing int.MaxValue instead.","Guard the caller code: skip the call when the limit is non-positive."],"exampleFix":"// before\ntok.EncodeToIds(text, considerPreTokenization: true, considerNormalization: true, maxTokenCount: budget); // budget == 0\n// after\ntok.EncodeToIds(text, true, true, maxTokenCount: Math.Max(1, budget));","handlingStrategy":"validation","validationCode":"if (maxTokenCount <= 0) throw new ArgumentOutOfRangeException(nameof(maxTokenCount));\nvar ids = tokenizer.EncodeToIds(text, true, true, maxTokenCount);","typeGuard":"bool IsValidTokenBudget(int n) => n > 0;","tryCatchPattern":"try { ids = tokenizer.EncodeToIds(text, true, true, maxTokenCount); }\ncatch (ArgumentOutOfRangeException) { ids = tokenizer.EncodeToIds(text); }","preventionTips":["Treat 0/negative as 'unlimited' by mapping to int.MaxValue at call sites","Clamp computed budgets with Math.Max(1, budget)","Avoid default-initialized ints as token limits"],"tags":["tokenizer","argument-out-of-range","dotnet"],"backgroundTag":"argument-out-of-range","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}