What causes "The max token count must be greater than 0." in dotnet/machinelearning?

Ensure the value passed to GetIndexByTokenCount/maxTokenCount is >= 1.

dotnet/machinelearning · error · ArgumentOutOfRangeException

The max token count must be greater than 0.

Error message

The max token count must be greater than 0.

What it means

The private LastIndexOf helper (used by GetIndexByTokenCount) throws ArgumentOutOfRangeException with the message 'The max token count must be greater than 0.' when maxTokenCount <= 0. Note the wording differs slightly from EncodeToIds/CountTokens but the rule is identical.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs:661

        /// If <paramRef name="fromEnd" /> is <see langword="true"/>, it represents the index of the first character to be included. In cases where no tokens fit, the result will be the text length; conversely,
        /// if all tokens fit, the result will be zero.
        /// </returns>
        protected override int GetIndexByTokenCount(string? text, ReadOnlySpan<char> textSpan, EncodeSettings settings, bool fromEnd, out string? normalizedText, out int tokenCount)
        {
            if (fromEnd)
            {
                return LastIndexOf(text, textSpan, settings.MaxTokenCount, settings.ConsiderNormalization, settings.ConsiderNormalization, out normalizedText, out tokenCount);
            }

            tokenCount = CountTokens(text, textSpan, settings.ConsiderPreTokenization, settings.ConsiderNormalization, out normalizedText, out int charsConsumed, settings.MaxTokenCount);
            return charsConsumed;
        }

        private int LastIndexOf(string? text, ReadOnlySpan<char> textSpan, int maxTokenCount, bool considerPreTokenization, bool considerNormalization, out string? normalizedText, out int tokenCount)
        {
            if (maxTokenCount <= 0)
            {
                throw new ArgumentOutOfRangeException(nameof(maxTokenCount), "The max token count must be greater than 0.");
            }

            if (string.IsNullOrEmpty(text) && textSpan.IsEmpty)
            {
                normalizedText = null;
                tokenCount = 0;
                return 0;
            }

            IEnumerable<(int Offset, int Length)>? splits = InitializeForEncoding(
                                                                text,
                                                                textSpan,
                                                                considerPreTokenization,
                                                                considerNormalization,
                                                                _normalizer,
                                                                _preTokenizer,
                                                                out normalizedText,
                                                                out ReadOnlySpan<char> textSpanToEncode,

View on GitHub (pinned to 7b76e69cf9)

Solutions

Ensure the value passed to GetIndexByTokenCount/maxTokenCount is >= 1.
Clamp: var mtc = Math.Max(1, requested);
Fix chunking loops so they terminate before the budget hits zero.
Unify validation and messages across the tokenizer API by validating at your own boundary first.

Example fix

// before
var idx = tok.GetIndexByTokenCount(text, 0, out int count);
// after
var idx = tok.GetIndexByTokenCount(text, Math.Max(1, budget), out int count);

Defensive patterns

Strategy: validation

Validate before calling

if (maxTokenCount <= 0)
    throw new ArgumentException("maxTokenCount must be >= 1", nameof(maxTokenCount));
var idx = tok.GetIndexByTokenCount(text, maxTokenCount, out int tokenCount);

Type guard

static bool IsValidMaxTokenCount(int v) => v > 0;

Try / catch

try { var idx = tok.GetIndexByTokenCount(text, mtc, out int count); }
catch (ArgumentOutOfRangeException)
{ mtc = Math.Max(1, mtc); var idx = tok.GetIndexByTokenCount(text, mtc, out int count); }

Prevention

In chunking/search-back loops, clamp the remaining budget to >= 1 before each call.
Initialize budgets from positive constants, never uninitialized fields.
Add a debug assertion maxTokenCount > 0 in your tokenizer helper layer.

When it happens

Trigger: Calling GetIndexByTokenCount with a maxTokenCount argument of 0 or a negative value; GetIndexByTokenCount forwards the caller's EncodeSettings.MaxTokenCount into LastIndexOf.

Common situations: Search-back logic computing a decreasing token budget that reaches 0; uninitialized settings; off-by-one in chunking loops.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/70bfd6bf89a371d7. Report an issue: GitHub.