dotnet/machinelearning · error · ArgumentNullException

throw new ArgumentNullException(nameof(tokenIds));

Error message

throw new ArgumentNullException(nameof(tokenIds));

What it means

BertTokenizer.BuildInputsWithSpecialTokens (list overload) rejects a null tokenIds argument with ArgumentNullException, since special tokens (CLS/SEP) cannot be wrapped around an absent sequence. The method documents this exception in its XML docs.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/BertTokenizer.cs:294

            }

            return ids;
        }

        /// <summary>
        /// Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BERT sequence has the following format:
        ///     - single sequence: `[CLS] tokenIds [SEP]`
        ///     - pair of sequences: `[CLS] tokenIds [SEP] additionalTokenIds [SEP]`
        /// </summary>
        /// <param name="tokenIds">List of IDs to which the special tokens will be added.</param>
        /// <param name="additionalTokenIds">Optional second list of IDs for sequence pairs.</param>
        /// <returns>The list of IDs with special tokens added.</returns>
        /// <exception cref="ArgumentNullException">When <paramref name="tokenIds"/> is null.</exception>
        public IReadOnlyList<int> BuildInputsWithSpecialTokens(IEnumerable<int> tokenIds, IEnumerable<int>? additionalTokenIds = null)
        {
            if (tokenIds is null)
            {
                throw new ArgumentNullException(nameof(tokenIds));
            }

            List<int> ids;

            if (tokenIds is ICollection<int> c1)
            {
                int capacity = c1.Count + 2;    // Add 2 for [CLS] and two [SEP] tokens.

                if (additionalTokenIds is not null)
                {
                    capacity += additionalTokenIds is ICollection<int> c2 ? c2.Count + 1 : c1.Count + 1;
                }

                ids = new(capacity) { ClassificationTokenId };
            }
            else
            {
                // slow path

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Initialize tokenIds to an empty collection before the call.
2. Handle the upstream null/error before building inputs.
3. Check the encode result for null before passing it on.

Example fix

// before
var inputs = tokenizer.BuildInputsWithSpecialTokens(tokenIds); // tokenIds null after failed encode
// after
if (tokenIds is null) throw new InvalidOperationException("tokenization failed");
var inputs = tokenizer.BuildInputsWithSpecialTokens(tokenIds);

if (tokenIds is null) throw new InvalidOperationException("tokenIds must be produced before building BERT inputs");

bool HasTokenIds(IEnumerable<int>? ids) => ids is not null;

try { inputs = tokenizer.BuildInputsWithSpecialTokens(tokenIds); }
catch (ArgumentNullException ex) when (ex.ParamName == "tokenIds") { /* handle failed tokenization upstream */ }

When it happens

Trigger: Calling BuildInputsWithSpecialTokens(IEnumerable<int> tokenIds, ...) with tokenIds == null, e.g. when an upstream encode step failed and returned null instead of a list.

Common situations: Preparing BERT model inputs where tokenization failures propagate null; deserialized request payloads missing the token IDs.

Related errors

• Value cannot be null. (Parameter 'input')
• columns
• rawType
• throw new ArgumentNullException(nameof(ids));
• argument {nameof(symbol)} should not be null.

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/11fadecf99448caf. Report an issue: GitHub.