dotnet/machinelearning · error · ArgumentOutOfRangeException

The maximum number of tokens must be greater than zero.

Error message

The maximum number of tokens must be greater than zero.

What it means

EncodeToIds enforces that settings.MaxTokenCount is strictly greater than zero and throws ArgumentOutOfRangeException otherwise. The parameter controls the token-budget cap during encoding; a zero or negative value has no valid meaning and indicates a caller bug.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs:494

                tokens.Add(new EncodedToken(EndOfSentenceId, EndOfSentenceToken, new Range(charsConsumed, charsConsumed)));
            }

            return new EncodeResults<EncodedToken> { Tokens = tokens, NormalizedText = normalizedText, CharsConsumed = charsConsumed };
        }

        /// <summary>
        /// Encodes input text to token Ids.
        /// </summary>
        /// <param name="text">The text to encode.</param>
        /// <param name="textSpan">The span of the text to encode which will be used if the <paramref name="text"/> is <see langword="null"/>.</param>
        /// <param name="settings">The settings used to encode the text.</param>
        /// <returns>The encoded results containing the list of encoded Ids.</returns>
        protected override EncodeResults<int> EncodeToIds(string? text, ReadOnlySpan<char> textSpan, EncodeSettings settings)
        {
            int maxTokenCount = settings.MaxTokenCount;
            if (maxTokenCount <= 0)
            {
                throw new ArgumentOutOfRangeException(nameof(settings.MaxTokenCount), "The maximum number of tokens must be greater than zero.");
            }

            if (string.IsNullOrEmpty(text) && textSpan.IsEmpty)
            {
                return new EncodeResults<int> { Tokens = [], NormalizedText = null, CharsConsumed = 0 };
            }

            IEnumerable<(int Offset, int Length)>? splits = InitializeForEncoding(
                                                                text,
                                                                textSpan,
                                                                settings.ConsiderPreTokenization,
                                                                settings.ConsiderNormalization,
                                                                _normalizer,
                                                                _preTokenizer,
                                                                out string? normalizedText,
                                                                out ReadOnlySpan<char> textSpanToEncode,
                                                                out _);

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Pass a positive maxTokenCount (e.g. the model's context length).
2. If you need no limit, pass int.MaxValue instead of 0.
3. Clamp computed budgets: Math.Max(1, budget - consumed).
4. Validate the config value > 0 before constructing EncodeSettings.

Example fix

// before
int max = config.MaxTokens; // 0 when unset
var ids = tokenizer.EncodeToIds(text, maxTokenCount: max);
// after
int max = config.MaxTokens > 0 ? config.MaxTokens : int.MaxValue;
var ids = tokenizer.EncodeToIds(text, maxTokenCount: max);

if (maxTokenCount <= 0) maxTokenCount = modelContextLength; // or int.MaxValue for no cap
var ids = tokenizer.EncodeToIds(text, maxTokenCount: maxTokenCount);

static int PositiveTokenLimit(int n) => n > 0 ? n : int.MaxValue;

try { var ids = tokenizer.EncodeToIds(text, settings); }
catch (ArgumentOutOfRangeException ex) when (ex.ParamName == "settings.MaxTokenCount") { settings.MaxTokenCount = int.MaxValue; var ids = tokenizer.EncodeToIds(text, settings); }

When it happens

Trigger: Calling tokenizer.EncodeToIds(text, maxTokenCount: 0), EncodeToIds(text, settings: new EncodeSettings { MaxTokenCount = -1 }), or default-initialized settings structs where MaxTokenCount was never set to a positive value.

Common situations: Reading MaxTokenCount from config that yields 0 (unset int / empty string parsed to 0); passing a user-supplied 'max length' of 0 meaning 'unlimited' instead of using int.MaxValue or no limit; arithmetic like totalBudget - alreadyUsed going negative.

Related errors

• The maximum number of tokens must be greater than zero.
• throw new ArgumentNullException(nameof(vocabStream));
• The beginning of sentence token '{beginningOfSentenceToken}'
• The end of sentence token '{endOfSentenceToken}' was not pre
• throw new ArgumentNullException(nameof(vocabulary));

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/0f4f29a214fbacb5. Report an issue: GitHub.