{"record":{"id":"f30b5d64cfa5dcff","repo":"dotnet/machinelearning","slug":"the-maximum-number-of-tokens-must-be-greater-than-f30b5d","errorCode":null,"errorMessage":"The maximum number of tokens must be greater than 0.","messagePattern":"The maximum number of tokens must be greater than 0\\.","errorType":"validation","errorClass":"ArgumentOutOfRangeException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceBpeModel.cs","lineNumber":340,"sourceCode":"                return [];\n            }\n\n            ReadOnlySpan<char> textToEncode;\n\n            if (considerNormalization && Normalizer is not null)\n            {\n                normalizedText = Normalizer.Normalize(text);\n                textToEncode = normalizedText.AsSpan();\n            }\n            else\n            {\n                normalizedText = null;\n                textToEncode = text;\n            }\n\n            if (maxTokenCount <= 0)\n            {\n                throw new ArgumentOutOfRangeException(nameof(maxTokenCount), \"The maximum number of tokens must be greater than 0.\");\n            }\n\n            List<int> ids = new();\n\n            if (SpecialTokensRegex is not null)\n            {\n                EncodeToIdsWithAddedToken(textToEncode, addBeginningOfSentence, addEndOfSentence, ids, out charsConsumed, maxTokenCount);\n            }\n            else\n            {\n                EncodeToIds(textToEncode, addBeginningOfSentence, addEndOfSentence, ids, out charsConsumed, maxTokenCount);\n            }\n\n            return ids;\n        }\n\n        private int EncodeToIdsWithAddedToken(ReadOnlySpan<char> text, bool addBeginOfSentence, bool addEndOfSentence, IList<int> accumulatedIds, out int charsConsumed, int maxTokens = int.MaxValue)\n        {","sourceCodeStart":322,"sourceCodeEnd":358,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceBpeModel.cs#L322-L358","documentation":"A second, redundant validation inside EncodeToIds re-checks maxTokenCount <= 0 and throws ArgumentOutOfRangeException with slightly different wording ('greater than 0'). It guards the internal continuation of the encoding path; callers normally hit the first check at the top of the same method.","triggerScenarios":"Any call path reaching this inner check with maxTokenCount <= 0 — practically the same as the entry check: calling EncodeToIds or overloads that route into it with 0 or a negative value.","commonSituations":"Same as the entry validation: computed limits dropping to zero/negative, or config values where 0 was intended as 'unlimited'.","solutions":["Pass a maxTokenCount >= 1 or omit the parameter to use int.MaxValue.","Translate a 0 'unlimited' sentinel to int.MaxValue before the call.","Clamp with Math.Max(1, maxTokenCount)."],"exampleFix":"// before\nvar ids = tokenizer.EncodeToIds(text, maxTokenCount: 0);\n// after\nvar ids = tokenizer.EncodeToIds(text, maxTokenCount: int.MaxValue); // unlimited, or a positive limit","handlingStrategy":"validation","validationCode":"if (maxTokenCount <= 0) throw new ArgumentOutOfRangeException(nameof(maxTokenCount));","typeGuard":"static bool IsValidMaxTokenCount(int n) => n > 0;","tryCatchPattern":"try { Encode(text, n); } catch (ArgumentOutOfRangeException) { Encode(text, int.MaxValue); }","preventionTips":["Reuse one shared clamp/validator for all maxTokenCount call sites","Never pass raw config values without a positivity check"],"tags":["argument-out-of-range","tokenization","dotnet"],"backgroundTag":"argument-out-of-range","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}