dotnet/machinelearning · error · ArgumentNullException

argument {nameof(symbol)} should not be null.

Error message

argument {nameof(symbol)} should not be null.

What it means

ReserveStringSymbolSlot validates that the symbol string is not null and throws ArgumentNullException naming the 'symbol' parameter. It registers special string symbols (like added tokens) in the tokenizer's symbol table.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/EnglishRobertaTokenizer.cs:1171

            return 0;
        }

        public int ConvertOccurrenceRankToId(int rank)
        {
            if ((uint)rank >= _symbols.Count)
            {
                return UnkIndex;
            }

            return _symbols[rank].Id;
        }

        private int ReserveStringSymbolSlot(string symbol, int defaultOccurrence = -1)
        {
            if (symbol is null)
            {
                throw new ArgumentNullException(nameof(symbol), $"argument {nameof(symbol)} should not be null.");
            }

            if (!_stringSymbolToIndexMapping.TryGetValue(symbol, out int idx))
            {
                idx = _symbols.Count;
                _symbols.Add((-1, defaultOccurrence));
                _stringSymbolToIndexMapping[symbol] = idx;
            }

            return idx;
        }

        public int AddSymbol(int id, int highOccurrenceScore)
        {
            if (!_idToIndex.TryGetValue(id, out int idx))
            {
                idx = _symbols.Count;
                _symbols.Add((id, highOccurrenceScore));

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Filter out null entries before passing token collections to the tokenizer: symbols.Where(s => s is not null).
2. Fix the source data (vocab/special-tokens file) so no token entry is null or empty.
3. If constructing programmatically, validate each symbol with ArgumentException.ThrowIfNull-style checks at your boundary.

Example fix

// before
var tokenizer = new EnglishRobertaTokenizer(vocab, merges, specialTokens: new string?[] { "<s>", null, "</s>" });
// after
var clean = new string?[] { "<s>", null, "</s>" }.Where(s => s is not null).Cast<string>().ToArray();
var tokenizer = new EnglishRobertaTokenizer(vocab, merges, specialTokens: clean);

if (symbols is not null)
    foreach (var s in symbols)
        if (s is null) throw new InvalidDataException("Special token list contains a null entry");

bool IsValidSymbol([NotNullWhen(true)] string? s) => !string.IsNullOrEmpty(s);
// use: var clean = symbols?.Where(IsValidSymbol).ToArray() ?? Array.Empty<string>();

try { InitTokenizer(specialTokens); }
catch (ArgumentNullException ex) when (ex.ParamName == "symbol") { throw new InvalidDataException("A token entry was null; check config/vocab data", ex); }

When it happens

Trigger: Calling internal symbol-registration paths (e.g., during construction with special-token lists) where one of the entries in the string-symbol collection is null.

Common situations: A special-tokens config or vocab file containing a null entry; deserialized token lists where missing entries became null; programmatic construction passing null in an array of added tokens.

Related errors

• Value cannot be null. (Parameter 'input')
• columns
• rawType
• throw new ArgumentNullException(nameof(vocabStream));
• throw new ArgumentNullException(nameof(ids));

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/5d1948230d6af2c4. Report an issue: GitHub.