dotnet/machinelearning · error · ArgumentNullException

vocab

Error message

vocab

What it means

GetPieceAtIndex requires a non-null vocabulary list; when it is called with a null pieces list it throws ArgumentNullException with the parameter name "vocab". This is an internal guard reached when a SentencePieceUnigramModel was constructed without a valid vocabulary.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceUnigramModel.cs:308

            }
        }

        private static void AddControlId(HashSet<int> set, int id)
        {
            if (id >= 0)
            {
                set.Add(id);
            }
        }

        private static int GetPieceCount(IReadOnlyList<(string Piece, float Score)>? pieces)
            => pieces?.Count ?? 0;

        private static string GetPieceAtIndex(IReadOnlyList<(string Piece, float Score)>? pieces, int index)
        {
            if (pieces is null)
            {
                throw new ArgumentNullException("vocab");
            }

            // A negative index means the model has no unknown token (HF permits a null unk_id). Return a cosmetic
            // default token that is never emitted (OOV is handled by byte fallback in that configuration).
            if (index < 0)
            {
                return "<unk>";
            }

            if (index >= pieces.Count)
            {
                throw new ArgumentOutOfRangeException("unkId", "unkId must be a valid index in the vocabulary.");
            }

            return pieces[index].Piece;
        }

        // Validates pieces is not null and unkId is in range; returns pieces unchanged.

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Ensure the vocabulary passed to the unigram model constructor is a non-null list of (Piece, Score) tuples.
2. Call ValidateVocab-style checks (pieces != null, unkId in range) before using the model.
3. Reload the model from a valid tokenizer.json/proto so the vocabulary is populated.

Example fix

// before
model = new SentencePieceUnigramModel(pieces: null, unkId: 0);
// after
model = new SentencePieceUnigramModel(pieces: loadedPieces, unkId: 0); // loadedPieces != null

// C# — ensure vocab is non-null before constructing/querying the model
if (pieces is null)
    throw new InvalidOperationException("Vocabulary must be loaded before creating SentencePieceUnigramModel.");

static bool HasVocab(IReadOnlyList<(string, float)>? p) => p is { Count: > 0 };

try { var piece = GetPieceAt(model, id); }
catch (ArgumentNullException ex) when (ex.ParamName == "vocab") { /* vocab was null — reload model */ }

When it happens

Trigger: Internal lookups (e.g. via SentencePieceUnigramModel special-token accessors) calling GetPieceAtIndex with pieces == null, which reflects a model instance created with a null vocabulary.

Common situations: Programmatically constructing the unigram model with a null/missing vocabulary collection and later querying token names; deserialization paths that left the vocab unset.

Related errors

• ArgumentNullException
• The BOS, EOS, or UNK token is not present in the vocabulary.
• The tokenizer.json model enables byte_fallback but does not
• unkId must be a valid index in the vocabulary.
• The vocabulary does not contain the required special token.

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/8af7e6ec6fc260c9. Report an issue: GitHub.