dotnet/machinelearning · error · ArgumentOutOfRangeException

The max token count must be greater than 0.

Error message

The max token count must be greater than 0.

What it means

LastIndexOf, the engine behind GetIndexByTokenCount, rejects maxTokenCount <= 0 with ArgumentOutOfRangeException. Since the method finds where a text can be split so it fits within maxTokenCount tokens, a zero or negative cap is meaningless and is rejected up front.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs:694

            }
            else
            {
                count = EncodeToIdsWithCache(textSpanToEncode, null, maxTokenCount, out charsConsumed, ref priorityQueue);
            }

            if (EndOfSentenceToken is not null && count < maxTokenCount)
            {
                count++;
            }

            return count;
        }

        private int LastIndexOf(string? text, ReadOnlySpan<char> textSpan, int maxTokenCount, bool considerPreTokenization, bool considerNormalization, out string? normalizedText, out int tokenCount)
        {
            if (maxTokenCount <= 0)
            {
                throw new ArgumentOutOfRangeException(nameof(maxTokenCount), "The max token count must be greater than 0.");
            }

            if (string.IsNullOrEmpty(text) && textSpan.IsEmpty)
            {
                normalizedText = null;
                tokenCount = 0;
                return 0;
            }

            IEnumerable<(int Offset, int Length)>? splits = InitializeForEncoding(
                                                                text,
                                                                textSpan,
                                                                considerPreTokenization,
                                                                considerNormalization,
                                                                _normalizer,
                                                                _preTokenizer,
                                                                out normalizedText,
                                                                out ReadOnlySpan<char> textSpanToEncode,

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Supply maxTokenCount >= 1.
2. Validate/normalize configured chunk sizes at load time (Math.Max(1, configured)).
3. Return an explicit 'nothing fits' result in caller logic instead of passing 0 to the tokenizer.

Example fix

// before
var res = tokenizer.GetIndexByTokenCount(text, considerPreTokenization: true, out _, maxTokenCount: chunkSize); // chunkSize == 0
// after
chunkSize = Math.Max(1, chunkSize);
var res = tokenizer.GetIndexByTokenCount(text, considerPreTokenization: true, out _, maxTokenCount: chunkSize);

if (chunkMaxTokens <= 0) throw new ArgumentException("Chunk size must be positive");

try { idx = tokenizer.GetIndexByTokenCount(text, true, out _, maxTokenCount: n); }
catch (ArgumentOutOfRangeException ex) when (ex.ParamName == "maxTokenCount") { /* fall back to chunking by chars */ }

When it happens

Trigger: Calling GetIndexByTokenCount (string or Span<char> overloads) on a BPETokenizer with maxTokenCount <= 0; also any wrapper that forwards a zero batch/segment limit.

Common situations: Chunking long documents where the per-chunk token budget was configured as 0, or an off-by-one that produced 0 after subtracting already-used tokens.

Related errors

• Either input_ids or inputs_embeds must be set
• Dimension must be divisible by 2
• Directory "{0}" does not exist.
• Unsupported pixel format
• Invalid width value.

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/725d94b29bfccb53. Report an issue: GitHub.