dotnet/machinelearning · error · ArgumentOutOfRangeException

The maximum number of tokens must be greater than zero.

Error message

The maximum number of tokens must be greater than zero.

What it means

Argument guard in TiktokenTokenizer.EncodeToIds (and its overloads): the maximum token count allowed for the encoded output must be greater than zero. Passing 0 or a negative value would make the encoding budget meaningless (no tokens could ever be produced), so the method validates the limit up front and throws before any text is encoded.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs:367

                                encodedTokens[i].Id,
                                encodedTokens[i].TokenLength == 0 ? string.Empty : text.Slice(encodedTokens[i].TokenIndex, encodedTokens[i].TokenLength).ToString(),
                                new Range(encodedTokens[i].TokenIndex + offset, encodedTokens[i].TokenIndex + offset + encodedTokens[i].TokenLength)));
            }
        }

        /// <summary>
        /// Encodes input text to token Ids.
        /// </summary>
        /// <param name="text">The text to encode.</param>
        /// <param name="textSpan">The span of the text to encode which will be used if the <paramref name="text"/> is <see langword="null"/>.</param>
        /// <param name="settings">The settings used to encode the text.</param>
        /// <returns>The encoded results containing the list of encoded Ids.</returns>
        protected override EncodeResults<int> EncodeToIds(string? text, ReadOnlySpan<char> textSpan, EncodeSettings settings)
        {
            int maxTokenCount = settings.MaxTokenCount;
            if (maxTokenCount <= 0)
            {
                throw new ArgumentOutOfRangeException(nameof(settings.MaxTokenCount), "The maximum number of tokens must be greater than zero.");
            }

            if (string.IsNullOrEmpty(text) && textSpan.IsEmpty)
            {
                return new EncodeResults<int> { NormalizedText = null, Tokens = [], CharsConsumed = 0 };
            }

            IEnumerable<(int Offset, int Length)>? splits = InitializeForEncoding(
                                                                text,
                                                                textSpan,
                                                                settings.ConsiderPreTokenization,
                                                                settings.ConsiderNormalization,
                                                                _normalizer,
                                                                _preTokenizer,
                                                                out string? normalizedText,
                                                                out ReadOnlySpan<char> textSpanToEncode,
                                                                out int charsConsumed);

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Pass MaxTokenCount = int.MaxValue (the default) when you want no limit instead of 0.
2. Clamp computed budgets to at least 1 before calling: Math.Max(1, remaining).
3. Ensure the settings object actually sets MaxTokenCount only when a valid positive limit is intended.
4. Guard inputs before encoding.

Example fix

// before
var r = tok.EncodeToIds(text, new EncodeSettings { MaxTokenCount = 0 });
// after
var r = tok.EncodeToIds(text, new EncodeSettings { MaxTokenCount = Math.Max(1, remaining) });

static int SafeMaxTokenCount(int requested) =>
    requested <= 0 ? int.MaxValue : requested;
// usage: settings.MaxTokenCount = SafeMaxTokenCount(computedBudget);

static bool IsValidMaxTokenCount(int? v) => v is null || v > 0;

try { var r = tok.EncodeToIds(text, settings); }
catch (ArgumentOutOfRangeException) { settings.MaxTokenCount = int.MaxValue; var r = tok.EncodeToIds(text, settings); }

When it happens

Trigger: Calling EncodeToIds (or the public EncodeToIds overloads) with an EncodeSettings whose MaxTokenCount is 0 or negative, e.g. 'new EncodeSettings { MaxTokenCount = 0 }'.

Common situations: Computing MaxTokenCount dynamically (e.g. promptWindowSize - promptTokens) so it underflows to 0 or negative; copying MaxTokenCount from an uninitialized int field; passing default structs misused as 'unlimited'.

Related errors

• The maximum number of tokens must be greater than zero.
• The max token count must be greater than 0.
• The maximum number of tokens must be greater than zero.
• failed to insert key: invalid null character
• Specified argument was out of the range of valid values. (Pa

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/45cd80696efc5031. Report an issue: GitHub.