dotnet/machinelearning · error · ArgumentOutOfRangeException

The maximum number of tokens must be greater than zero.

Error message

The maximum number of tokens must be greater than zero.

What it means

EncodeToIds validates that maxTokenCount is a positive integer before encoding; zero or negative values are meaningless caps and throw ArgumentOutOfRangeException naming the parameter. This is a standard guard on the public EncodeToIds overloads accepting string or ReadOnlySpan<char>.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs:582

        {
            return EncodeToIds(null, text, addPrefixSpace, addBeginningOfSentence, addEndOfSentence, considerPreTokenization, considerNormalization, out normalizedText, out charsConsumed, maxTokenCount);
        }

        private IReadOnlyList<int> EncodeToIds(
                                    string? text,
                                    scoped ReadOnlySpan<char> textSpan,
                                    bool addPrefixSpace,
                                    bool addBeginningOfSentence,
                                    bool addEndOfSentence,
                                    bool considerPreTokenization,
                                    bool considerNormalization,
                                    out string? normalizedText,
                                    out int charsConsumed,
                                    int maxTokenCount = int.MaxValue)
        {
            if (maxTokenCount <= 0)
            {
                throw new ArgumentOutOfRangeException(nameof(maxTokenCount), "The maximum number of tokens must be greater than zero.");
            }

            if (string.IsNullOrEmpty(text) && textSpan.IsEmpty)
            {
                charsConsumed = 0;
                normalizedText = null;
                return [];
            }

            char[]? mutatedInputText = null;

            try
            {
                Span<char> mutatedInputSpan = stackalloc char[BufferLength];
                scoped ReadOnlySpan<char> textSpanToEncode;
                IEnumerable<(int Offset, int Length)>? splits;
                if (addPrefixSpace)
                {

View on GitHub (pinned to 7b76e69cf9)

Solutions

  1. Pass maxTokenCount > 0, or omit it to use the int.MaxValue default.
  2. Clamp computed limits: Math.Max(1, computedLimit) before calling.
  3. Treat 0 as 'no limit' at the call site by passing int.MaxValue instead.
  4. Guard the caller code: skip the call when the limit is non-positive.

Example fix

// before
tok.EncodeToIds(text, considerPreTokenization: true, considerNormalization: true, maxTokenCount: budget); // budget == 0
// after
tok.EncodeToIds(text, true, true, maxTokenCount: Math.Max(1, budget));
Defensive patterns

Strategy: validation

Validate before calling

if (maxTokenCount <= 0) throw new ArgumentOutOfRangeException(nameof(maxTokenCount));
var ids = tokenizer.EncodeToIds(text, true, true, maxTokenCount);

Type guard

bool IsValidTokenBudget(int n) => n > 0;

Try / catch

try { ids = tokenizer.EncodeToIds(text, true, true, maxTokenCount); }
catch (ArgumentOutOfRangeException) { ids = tokenizer.EncodeToIds(text); }

Prevention

When it happens

Trigger: Calling tokenizer.EncodeToIds(text or span, ..., maxTokenCount) with maxTokenCount = 0 or negative, often a default-initialized int or a computed limit like text.Length when the text is empty.

Common situations: A caller-supplied 'budget' variable that was never set (defaults to 0), or off-by-one arithmetic producing a non-positive cap.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/d8072de5795733ad. Report an issue: GitHub.