What causes "The maximum number of tokens must be greater than 0." in dotnet/machinelearning?

Pass a maxTokenCount >= 1 or omit the parameter to use int.MaxValue.

dotnet/machinelearning · error · ArgumentOutOfRangeException

The maximum number of tokens must be greater than 0.

Error message

The maximum number of tokens must be greater than 0.

What it means

A second, redundant validation inside EncodeToIds re-checks maxTokenCount <= 0 and throws ArgumentOutOfRangeException with slightly different wording ('greater than 0'). It guards the internal continuation of the encoding path; callers normally hit the first check at the top of the same method.

Solutions

Pass a maxTokenCount >= 1 or omit the parameter to use int.MaxValue.
Translate a 0 'unlimited' sentinel to int.MaxValue before the call.
Clamp with Math.Max(1, maxTokenCount).

Example fix

// before
var ids = tokenizer.EncodeToIds(text, maxTokenCount: 0);
// after
var ids = tokenizer.EncodeToIds(text, maxTokenCount: int.MaxValue); // unlimited, or a positive limit

Defensive patterns

Strategy: validation

Validate before calling

if (maxTokenCount <= 0) throw new ArgumentOutOfRangeException(nameof(maxTokenCount));

Type guard

static bool IsValidMaxTokenCount(int n) => n > 0;

Try / catch

try { Encode(text, n); } catch (ArgumentOutOfRangeException) { Encode(text, int.MaxValue); }

Prevention

Reuse one shared clamp/validator for all maxTokenCount call sites
Never pass raw config values without a positivity check

When it happens

Trigger: Any call path reaching this inner check with maxTokenCount <= 0 — practically the same as the entry check: calling EncodeToIds or overloads that route into it with 0 or a negative value.

Common situations: Same as the entry validation: computed limits dropping to zero/negative, or config values where 0 was intended as 'unlimited'.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/f30b5d64cfa5dcff. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceBpeModel.cs:340

                return [];
            }

            ReadOnlySpan<char> textToEncode;

            if (considerNormalization && Normalizer is not null)
            {
                normalizedText = Normalizer.Normalize(text);
                textToEncode = normalizedText.AsSpan();
            }
            else
            {
                normalizedText = null;
                textToEncode = text;
            }

            if (maxTokenCount <= 0)
            {
                throw new ArgumentOutOfRangeException(nameof(maxTokenCount), "The maximum number of tokens must be greater than 0.");
            }

            List<int> ids = new();

            if (SpecialTokensRegex is not null)
            {
                EncodeToIdsWithAddedToken(textToEncode, addBeginningOfSentence, addEndOfSentence, ids, out charsConsumed, maxTokenCount);
            }
            else
            {
                EncodeToIds(textToEncode, addBeginningOfSentence, addEndOfSentence, ids, out charsConsumed, maxTokenCount);
            }

            return ids;
        }

        private int EncodeToIdsWithAddedToken(ReadOnlySpan<char> text, bool addBeginOfSentence, bool addEndOfSentence, IList<int> accumulatedIds, out int charsConsumed, int maxTokens = int.MaxValue)
        {

View on GitHub (pinned to 7b76e69cf9)