dotnet/machinelearning · error · ArgumentNullException

throw new ArgumentNullException(nameof(ids));

Error message

throw new ArgumentNullException(nameof(ids));

What it means

The string-returning Decode overload requires a non-null sequence of token IDs. A null ids argument is rejected immediately with ArgumentNullException because there is no meaningful decode target. Empty sequences are allowed and decode to an empty string.

Solutions

1. Ensure the IDs collection is initialized (e.g. Array.Empty<int>()) before decoding.
2. Check the source of the IDs for a null-producing error path and handle it before decoding.
3. Coalesce null to an empty sequence if decoding nothing is acceptable.

Example fix

// before
string text = tokenizer.Decode(ids, considerSpecialTokens: false); // ids is null
// after
string text = tokenizer.Decode(ids ?? (IEnumerable<int>)Array.Empty<int>(), considerSpecialTokens: false);

if (ids is null) throw new InvalidOperationException("No token IDs to decode; encoding step failed.");

bool HasIds(IEnumerable<int>? ids) => ids is not null;

try { text = tokenizer.Decode(ids, false); }
catch (ArgumentNullException ex) when (ex.ParamName == "ids") { text = string.Empty; /* surface upstream failure */ }

When it happens

Trigger: Calling BpeTokenizer.Decode(IEnumerable<int> ids, bool considerSpecialTokens) with ids == null, typically when the variable holding generated token IDs was never assigned or a decode path ran after a failed encode returned null.

Common situations: Pipelines where model output IDs are null on error but the code decodes unconditionally; deserialized objects with a null Ids property.

Related errors

• argument should not be null.
• columns
• Invalid input stream contents
• Invalid path
• rawType

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/01212530ada7c273. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs:785

        /// <summary>
        /// Decode the given ids, back to a String.
        /// </summary>
        /// <param name="ids">The list of ids that we want to decode.</param>
        /// <returns>The decoded string.</returns>
        public override string Decode(IEnumerable<int> ids) => Decode(ids, considerSpecialTokens: true);

        /// <summary>
        /// Decode the given ids, back to a String.
        /// </summary>
        /// <param name="ids">The list of ids that we want to decode.</param>
        /// <param name="considerSpecialTokens">Indicate whether to consider special tokens or not.</param>
        /// <returns>The decoded string.</returns>
        public string Decode(IEnumerable<int> ids, bool considerSpecialTokens)
        {
            if (ids is null)
            {
                throw new ArgumentNullException(nameof(ids));
            }

            if (ByteLevel)
            {
                return DecodeByteLevel(ids, considerSpecialTokens);
            }

            ValueStringBuilder sb = new ValueStringBuilder();

            foreach (int id in ids)
            {
                if (_specialTokensReverse?.TryGetValue(id, out string? token) is true)
                {
                    if (considerSpecialTokens)
                    {
                        sb.Append(token);
                    }
                    continue;

View on GitHub (pinned to 7b76e69cf9)