dotnet/machinelearning · error · ArgumentOutOfRangeException

unkId must be a valid index in the vocabulary.

Error message

unkId must be a valid index in the vocabulary.

What it means

GetPieceAtIndex indexes into the vocabulary to fetch a piece string by id; an index >= pieces.Count is out of range, so it throws ArgumentOutOfRangeException naming unkId with the message that the id must be a valid vocabulary index. A negative index is allowed (model without unk token) but ids at or beyond the vocabulary size are not.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceUnigramModel.cs:320

            => pieces?.Count ?? 0;

        private static string GetPieceAtIndex(IReadOnlyList<(string Piece, float Score)>? pieces, int index)
        {
            if (pieces is null)
            {
                throw new ArgumentNullException("vocab");
            }

            // A negative index means the model has no unknown token (HF permits a null unk_id). Return a cosmetic
            // default token that is never emitted (OOV is handled by byte fallback in that configuration).
            if (index < 0)
            {
                return "<unk>";
            }

            if (index >= pieces.Count)
            {
                throw new ArgumentOutOfRangeException("unkId", "unkId must be a valid index in the vocabulary.");
            }

            return pieces[index].Piece;
        }

        // Validates pieces is not null and unkId is in range; returns pieces unchanged.
        private static IReadOnlyList<(string Piece, float Score)> ValidateVocab(
            IReadOnlyList<(string Piece, float Score)>? pieces, int unkId)
        {
            if (pieces is null)
            {
                throw new ArgumentNullException("vocab");
            }

            if ((uint)unkId >= (uint)pieces.Count)
            {
                throw new ArgumentOutOfRangeException("unkId", "unkId must be a valid index in the vocabulary.");
            }

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Pass an unkId (and any special ids) within [0, pieces.Count - 1] when constructing the model.
2. Rebuild the vocabulary/ids from the same source so counts match.
3. Validate ids before construction: 0 <= unkId < vocab.Count.
4. If the model has no unk token, pass -1 rather than an out-of-range id.

Example fix

// before
new SentencePieceUnigramModel(pieces, unkId: pieces.Count); // out of range
// after
new SentencePieceUnigramModel(pieces, unkId: 0); // 0 <= unkId < pieces.Count

// C# — bounds-check ids before querying pieces
static string SafePieceAt(IReadOnlyList<(string Piece, float)> pieces, int index) =>
    index < 0 ? "<unk>" :
    (uint)index < (uint)pieces.Count ? pieces[index].Piece :
    throw new ArgumentOutOfRangeException(nameof(index));

static bool IsValidId(int id, int vocabCount) => (uint)id < (uint)vocabCount;

try { var piece = model.GetPieceAtIndex(id); }
catch (ArgumentOutOfRangeException ex) when (ex.ParamName == "unkId") { /* id out of vocab range — use matched vocab */ }

When it happens

Trigger: Querying a piece by id where the id equals or exceeds the vocabulary size — e.g. an unkId (or other special id) pointing past the end of the pieces list when accessed through SentencePieceUnigramModel.

Common situations: Models whose trainer-spec ids were not updated after the vocabulary shrank; ids loaded from a different model version than the vocabulary; hand-built vocabularies with mismatched counts.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• The BOS, EOS, or UNK token is not present in the vocabulary.
• The tokenizer.json model enables byte_fallback but does not
• vocab
• The vocabulary does not contain the required special token.
• Specified argument was out of the range of valid values. (Pa

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/9303deca0cb9bc34. Report an issue: GitHub.