{"record":{"id":"9303deca0cb9bc34","repo":"dotnet/machinelearning","slug":"unkid-must-be-a-valid-index-in-the-vocabulary","errorCode":null,"errorMessage":"unkId must be a valid index in the vocabulary.","messagePattern":"unkId must be a valid index in the vocabulary\\.","errorType":"exception","errorClass":"ArgumentOutOfRangeException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceUnigramModel.cs","lineNumber":320,"sourceCode":"            => pieces?.Count ?? 0;\n\n        private static string GetPieceAtIndex(IReadOnlyList<(string Piece, float Score)>? pieces, int index)\n        {\n            if (pieces is null)\n            {\n                throw new ArgumentNullException(\"vocab\");\n            }\n\n            // A negative index means the model has no unknown token (HF permits a null unk_id). Return a cosmetic\n            // default token that is never emitted (OOV is handled by byte fallback in that configuration).\n            if (index < 0)\n            {\n                return \"<unk>\";\n            }\n\n            if (index >= pieces.Count)\n            {\n                throw new ArgumentOutOfRangeException(\"unkId\", \"unkId must be a valid index in the vocabulary.\");\n            }\n\n            return pieces[index].Piece;\n        }\n\n        // Validates pieces is not null and unkId is in range; returns pieces unchanged.\n        private static IReadOnlyList<(string Piece, float Score)> ValidateVocab(\n            IReadOnlyList<(string Piece, float Score)>? pieces, int unkId)\n        {\n            if (pieces is null)\n            {\n                throw new ArgumentNullException(\"vocab\");\n            }\n\n            if ((uint)unkId >= (uint)pieces.Count)\n            {\n                throw new ArgumentOutOfRangeException(\"unkId\", \"unkId must be a valid index in the vocabulary.\");\n            }","sourceCodeStart":302,"sourceCodeEnd":338,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceUnigramModel.cs#L302-L338","documentation":"GetPieceAtIndex indexes into the vocabulary to fetch a piece string by id; an index >= pieces.Count is out of range, so it throws ArgumentOutOfRangeException naming unkId with the message that the id must be a valid vocabulary index. A negative index is allowed (model without unk token) but ids at or beyond the vocabulary size are not.","triggerScenarios":"Querying a piece by id where the id equals or exceeds the vocabulary size — e.g. an unkId (or other special id) pointing past the end of the pieces list when accessed through SentencePieceUnigramModel.","commonSituations":"Models whose trainer-spec ids were not updated after the vocabulary shrank; ids loaded from a different model version than the vocabulary; hand-built vocabularies with mismatched counts.","solutions":["Pass an unkId (and any special ids) within [0, pieces.Count - 1] when constructing the model.","Rebuild the vocabulary/ids from the same source so counts match.","Validate ids before construction: 0 <= unkId < vocab.Count.","If the model has no unk token, pass -1 rather than an out-of-range id."],"exampleFix":"// before\nnew SentencePieceUnigramModel(pieces, unkId: pieces.Count); // out of range\n// after\nnew SentencePieceUnigramModel(pieces, unkId: 0); // 0 <= unkId < pieces.Count","handlingStrategy":"validation","validationCode":"// C# — bounds-check ids before querying pieces\nstatic string SafePieceAt(IReadOnlyList<(string Piece, float)> pieces, int index) =>\n    index < 0 ? \"<unk>\" :\n    (uint)index < (uint)pieces.Count ? pieces[index].Piece :\n    throw new ArgumentOutOfRangeException(nameof(index));","typeGuard":"static bool IsValidId(int id, int vocabCount) => (uint)id < (uint)vocabCount;","tryCatchPattern":"try { var piece = model.GetPieceAtIndex(id); }\ncatch (ArgumentOutOfRangeException ex) when (ex.ParamName == \"unkId\") { /* id out of vocab range — use matched vocab */ }","preventionTips":["Keep ids and vocabulary from the same model version together.","Derive special ids from TrainerSpec, never hard-code them.","Bounds-check ids at configuration load time.","After vocab edits, re-run a full id-range audit."],"tags":["sentencepiece","argument-out-of-range","vocabulary"],"backgroundTag":"value-out-of-range","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}