{"record":{"id":"67aa6d7cc5bda622","repo":"dotnet/machinelearning","slug":"the-maximum-number-of-tokens-must-be-greater-than-67aa6d","errorCode":null,"errorMessage":"The maximum number of tokens must be greater than zero.","messagePattern":"The maximum number of tokens must be greater than zero\\.","errorType":"validation","errorClass":"ArgumentOutOfRangeException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceBpeModel.cs","lineNumber":286,"sourceCode":"                if (id.Type != (byte)ModelProto.Types.SentencePiece.Types.Type.Unused ||\n                    revMerge is null ||\n                    !revMerge.TryGetValue((pieceSpan.Index, pieceSpan.Length), out (int LeftIndex, int LeftLen, int RightIndex, int RightLen) merge))\n                {\n                    tokens.Add(new EncodedToken(id.Id, text.Slice(pieceSpan.Index, pieceSpan.Length).ToString(), new Range(pieceSpan.Index, pieceSpan.Index + pieceSpan.Length)));\n                    return;\n                }\n\n                Segment((merge.LeftIndex, merge.LeftLen), text);\n                Segment((merge.RightIndex, merge.RightLen), text);\n            }\n        }\n\n        public override IReadOnlyList<int> EncodeToIds(string? text, ReadOnlySpan<char> textSpan, bool addBeginningOfSentence, bool addEndOfSentence, bool considerNormalization,\n                                        out string? normalizedText, out int charsConsumed, int maxTokenCount = int.MaxValue)\n        {\n            if (maxTokenCount <= 0)\n            {\n                throw new ArgumentOutOfRangeException(nameof(maxTokenCount), \"The maximum number of tokens must be greater than zero.\");\n            }\n\n            if (string.IsNullOrEmpty(text) && textSpan.IsEmpty)\n            {\n                normalizedText = null;\n                charsConsumed = 0;\n                return [];\n            }\n\n            return EncodeToIds(text is null ? textSpan : text.AsSpan(), addBeginningOfSentence, addEndOfSentence, considerNormalization, out normalizedText, out charsConsumed, maxTokenCount);\n        }\n\n        /// <summary>\n        /// Encodes input text to token Ids up to maximum number of tokens.\n        /// </summary>\n        /// <param name=\"text\">The text to encode.</param>\n        /// <param name=\"addBeginningOfSentence\">Indicate emitting the beginning of sentence token during the encoding.</param>\n        /// <param name=\"addEndOfSentence\">Indicate emitting the end of sentence token during the encoding.</param>","sourceCodeStart":268,"sourceCodeEnd":304,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceBpeModel.cs#L268-L304","documentation":"SentencePieceBpeModel.EncodeToIds validates maxTokenCount and throws ArgumentOutOfRangeException when it is zero or negative, since at least one token must be allowed. maxTokenCount defaults to int.MaxValue, so this only happens when explicitly set.","triggerScenarios":"Calling EncodeToIds with maxTokenCount: 0 or a negative value, often from a computed limit (e.g. budget left = maxTokens - usedTokens which hit 0 or went negative).","commonSituations":"Config-driven max-length settings that are 0 or unset; arithmetic computing a remaining budget that underflows; UI inputs defaulting to 0.","solutions":["Pass maxTokenCount >= 1 (or omit it to use the int.MaxValue default)","Clamp computed budgets: Math.Max(1, remaining)","Validate configuration values for max token length at startup"],"exampleFix":"// before\nvar ids = tok.EncodeToIds(text, maxTokenCount: remaining); // remaining == 0\n// after\nvar ids = tok.EncodeToIds(text, maxTokenCount: Math.Max(1, remaining));","handlingStrategy":"validation","validationCode":"if (maxTokenCount <= 0) throw new InvalidOperationException(\"maxTokenCount must be at least 1.\");","typeGuard":null,"tryCatchPattern":"try { var ids = tok.EncodeToIds(text, maxTokenCount: limit); } catch (ArgumentOutOfRangeException ex) { log.LogError(ex, \"maxTokenCount must be > 0\"); throw; }","preventionTips":["Clamp computed budgets to at least 1","Validate configured max-length values at startup","Rely on the int.MaxValue default unless truncation is intended"],"tags":["tokenizers","argument-out-of-range","encode"],"backgroundTag":"argument-out-of-range","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}