dotnet/machinelearning · error · ArgumentOutOfRangeException

Index should be non-negative, got

Error message

Index should be non-negative, got {idx}.

What it means

The Symbols indexer returns the token ID at a given index, or UnkIndex for out-of-range (but non-negative) indices; a negative index is invalid and throws ArgumentOutOfRangeException with the offending value in the message.

Solutions

1. Check the index is >= 0 before indexing; treat negative values as 'token not found' and use the unknown token ID instead.
2. Fix the lookup that produced the negative value to use TryGetValue semantics rather than a -1 sentinel.
3. Note: out-of-range non-negative indices already fall back to UnkIndex, so only negativity needs guarding.

Example fix

// before
int id = tokenizer.Symbols[lookupResult]; // lookupResult may be -1
// after
int id = lookupResult >= 0 ? tokenizer.Symbols[lookupResult] : tokenizer.Symbols[tokenizer.UnknownTokenId]; // or UnkIndex fallback

if (index < 0) throw new InvalidOperationException("Symbol index must be non-negative");
int id = tokenizer.Symbols[index];

bool TryGetSymbolId(System.Collections.Generic.IReadOnlyList<int> lookup, string token, out int id)
{ id = lookup.TryGetValue(token, out var idx) && idx >= 0 ? idx : -1; return id >= 0; }

try { id = symbols[index]; }
catch (ArgumentOutOfRangeException ex) when (ex.ParamName == "idx") { id = unkIndex; log($"Negative symbol index {ex.ActualValue}"); }

When it happens

Trigger: Indexing tokenizer.Symbols (or the RobertaSymbol collection) with a negative int, typically from a lookup that returned -1 as 'not found' and the result being used as an index.

Common situations: Using a TryGetValue-style result of -1 as an index; arithmetic like lastIndex-1 when the collection is empty; mapping a vocab lookup miss (-1) directly into the symbol table.

Related errors

• The max token count must be greater than 0.
• The maximum number of tokens must be greater than zero.
• argument should not be null.
• Cannot read the file Merge file.
• Cannot read the file Merge file.

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/107fab95f6f1e2f8. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.ML.Tokenizers/Model/EnglishRobertaTokenizer.cs:1210

            return idx;
        }

        public int AddMaskSymbol(string mask = "<mask>")
        {
            MaskWord = mask;
            MaskIndex = ReserveStringSymbolSlot(mask, 1);
            return MaskIndex;
        }

        /// <exception cref="ArgumentOutOfRangeException">`idx` is negative.</exception>
        public int this[int idx]
        {
            get
            {
                if (idx < 0)
                {
                    throw new ArgumentOutOfRangeException(nameof(idx), $"Index should be non-negative, got {idx}.");
                }

                return idx < _symbols.Count ? _symbols[idx].Id : UnkIndex;
            }
        }

        public int Count => _symbols.Count;

        public bool Equals(HighestOccurrenceMapping other) => _idToIndex.SequenceEqual(other._idToIndex);

        public bool Contains(string symbol) => symbol != null && _stringSymbolToIndexMapping.ContainsKey(symbol);

        public bool Contains(int id) => _idToIndex.ContainsKey(id);

        /// <exception cref="ArgumentNullException">`symbol` is `null`.</exception>
        public int IndexOf(int id) => _idToIndex.ContainsKey(id) ? _idToIndex[id] : UnkIndex;

        /// <summary>

View on GitHub (pinned to 7b76e69cf9)