{"record":{"id":"5d1948230d6af2c4","repo":"dotnet/machinelearning","slug":"argument-nameof-symbol-should-not-be-null","errorCode":null,"errorMessage":"argument {nameof(symbol)} should not be null.","messagePattern":"argument (.+?) should not be null\\.","errorType":"exception","errorClass":"ArgumentNullException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/EnglishRobertaTokenizer.cs","lineNumber":1171,"sourceCode":"\n            return 0;\n        }\n\n        public int ConvertOccurrenceRankToId(int rank)\n        {\n            if ((uint)rank >= _symbols.Count)\n            {\n                return UnkIndex;\n            }\n\n            return _symbols[rank].Id;\n        }\n\n        private int ReserveStringSymbolSlot(string symbol, int defaultOccurrence = -1)\n        {\n            if (symbol is null)\n            {\n                throw new ArgumentNullException(nameof(symbol), $\"argument {nameof(symbol)} should not be null.\");\n            }\n\n            if (!_stringSymbolToIndexMapping.TryGetValue(symbol, out int idx))\n            {\n                idx = _symbols.Count;\n                _symbols.Add((-1, defaultOccurrence));\n                _stringSymbolToIndexMapping[symbol] = idx;\n            }\n\n            return idx;\n        }\n\n        public int AddSymbol(int id, int highOccurrenceScore)\n        {\n            if (!_idToIndex.TryGetValue(id, out int idx))\n            {\n                idx = _symbols.Count;\n                _symbols.Add((id, highOccurrenceScore));","sourceCodeStart":1153,"sourceCodeEnd":1189,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/EnglishRobertaTokenizer.cs#L1153-L1189","documentation":"ReserveStringSymbolSlot validates that the symbol string is not null and throws ArgumentNullException naming the 'symbol' parameter. It registers special string symbols (like added tokens) in the tokenizer's symbol table.","triggerScenarios":"Calling internal symbol-registration paths (e.g., during construction with special-token lists) where one of the entries in the string-symbol collection is null.","commonSituations":"A special-tokens config or vocab file containing a null entry; deserialized token lists where missing entries became null; programmatic construction passing null in an array of added tokens.","solutions":["Filter out null entries before passing token collections to the tokenizer: symbols.Where(s => s is not null).","Fix the source data (vocab/special-tokens file) so no token entry is null or empty.","If constructing programmatically, validate each symbol with ArgumentException.ThrowIfNull-style checks at your boundary."],"exampleFix":"// before\nvar tokenizer = new EnglishRobertaTokenizer(vocab, merges, specialTokens: new string?[] { \"<s>\", null, \"</s>\" });\n// after\nvar clean = new string?[] { \"<s>\", null, \"</s>\" }.Where(s => s is not null).Cast<string>().ToArray();\nvar tokenizer = new EnglishRobertaTokenizer(vocab, merges, specialTokens: clean);","handlingStrategy":"type-guard","validationCode":"if (symbols is not null)\n    foreach (var s in symbols)\n        if (s is null) throw new InvalidDataException(\"Special token list contains a null entry\");","typeGuard":"bool IsValidSymbol([NotNullWhen(true)] string? s) => !string.IsNullOrEmpty(s);\n// use: var clean = symbols?.Where(IsValidSymbol).ToArray() ?? Array.Empty<string>();","tryCatchPattern":"try { InitTokenizer(specialTokens); }\ncatch (ArgumentNullException ex) when (ex.ParamName == \"symbol\") { throw new InvalidDataException(\"A token entry was null; check config/vocab data\", ex); }","preventionTips":["Filter null/empty entries out of special-token collections before construction.","Validate deserialized token lists (JSON missing entries become null).","Never populate token arrays from unvalidated external config without checks.","Test tokenizer construction with malformed token lists."],"tags":["null-argument","tokenizer","csharp"],"backgroundTag":"null-argument","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}