{"record":{"id":"725d94b29bfccb53","repo":"dotnet/machinelearning","slug":"the-max-token-count-must-be-greater-than-0","errorCode":null,"errorMessage":"The max token count must be greater than 0.","messagePattern":"The max token count must be greater than 0\\.","errorType":"validation","errorClass":"ArgumentOutOfRangeException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs","lineNumber":694,"sourceCode":"            }\n            else\n            {\n                count = EncodeToIdsWithCache(textSpanToEncode, null, maxTokenCount, out charsConsumed, ref priorityQueue);\n            }\n\n            if (EndOfSentenceToken is not null && count < maxTokenCount)\n            {\n                count++;\n            }\n\n            return count;\n        }\n\n        private int LastIndexOf(string? text, ReadOnlySpan<char> textSpan, int maxTokenCount, bool considerPreTokenization, bool considerNormalization, out string? normalizedText, out int tokenCount)\n        {\n            if (maxTokenCount <= 0)\n            {\n                throw new ArgumentOutOfRangeException(nameof(maxTokenCount), \"The max token count must be greater than 0.\");\n            }\n\n            if (string.IsNullOrEmpty(text) && textSpan.IsEmpty)\n            {\n                normalizedText = null;\n                tokenCount = 0;\n                return 0;\n            }\n\n            IEnumerable<(int Offset, int Length)>? splits = InitializeForEncoding(\n                                                                text,\n                                                                textSpan,\n                                                                considerPreTokenization,\n                                                                considerNormalization,\n                                                                _normalizer,\n                                                                _preTokenizer,\n                                                                out normalizedText,\n                                                                out ReadOnlySpan<char> textSpanToEncode,","sourceCodeStart":676,"sourceCodeEnd":712,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs#L676-L712","documentation":"LastIndexOf, the engine behind GetIndexByTokenCount, rejects maxTokenCount <= 0 with ArgumentOutOfRangeException. Since the method finds where a text can be split so it fits within maxTokenCount tokens, a zero or negative cap is meaningless and is rejected up front.","triggerScenarios":"Calling GetIndexByTokenCount (string or Span<char> overloads) on a BPETokenizer with maxTokenCount <= 0; also any wrapper that forwards a zero batch/segment limit.","commonSituations":"Chunking long documents where the per-chunk token budget was configured as 0, or an off-by-one that produced 0 after subtracting already-used tokens.","solutions":["Supply maxTokenCount >= 1.","Validate/normalize configured chunk sizes at load time (Math.Max(1, configured)).","Return an explicit 'nothing fits' result in caller logic instead of passing 0 to the tokenizer."],"exampleFix":"// before\nvar res = tokenizer.GetIndexByTokenCount(text, considerPreTokenization: true, out _, maxTokenCount: chunkSize); // chunkSize == 0\n// after\nchunkSize = Math.Max(1, chunkSize);\nvar res = tokenizer.GetIndexByTokenCount(text, considerPreTokenization: true, out _, maxTokenCount: chunkSize);","handlingStrategy":"validation","validationCode":"if (chunkMaxTokens <= 0) throw new ArgumentException(\"Chunk size must be positive\");","typeGuard":null,"tryCatchPattern":"try { idx = tokenizer.GetIndexByTokenCount(text, true, out _, maxTokenCount: n); }\ncatch (ArgumentOutOfRangeException ex) when (ex.ParamName == \"maxTokenCount\") { /* fall back to chunking by chars */ }","preventionTips":["Sanity-check chunk-size config on load.","Treat a 0 remaining budget as 'nothing fits' before calling.","Write unit tests for chunk-size edge values 0 and 1."],"tags":["csharp","tokenizers","argument-validation"],"backgroundTag":"argument-out-of-range","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}