{"record":{"id":"c8c3f601260fdcb9","repo":"dotnet/machinelearning","slug":"the-bos-eos-or-unk-token-is-not-present-in-the-v","errorCode":null,"errorMessage":"The BOS, EOS, or UNK token is not present in the vocabulary.","messagePattern":"The BOS, EOS, or UNK token is not present in the vocabulary\\.","errorType":"exception","errorClass":"ArgumentException","httpStatus":null,"severity":"critical","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceUnigramModel.cs","lineNumber":38,"sourceCode":"    {\n        private readonly SortedDictionary<string, int> _vocab;\n        private readonly (string Piece, float Score, ModelProto.Types.SentencePiece.Types.Type Type)[] _vocabReverse;\n        private readonly DoubleArrayTrie _trie;\n        private readonly float _minScore;\n        private readonly float _maxScore;\n        private readonly (int Id, string Token)[] _prefixTokens;\n        private readonly (int Id, string Token)[] _suffixTokens;\n        private const float UnkPenalty = 10.0f;\n\n        public SentencePieceUnigramModel(ModelProto modelProto, bool addBos, bool addEos, IReadOnlyDictionary<string, int>? specialTokens = null) : base(modelProto, addBos, addEos, specialTokens)\n        {\n            _vocab = new SortedDictionary<string, int>(OrdinalUtf8StringComparer.Instance);\n\n            if (modelProto.TrainerSpec.BosId >= modelProto.Pieces.Count ||\n                modelProto.TrainerSpec.EosId >= modelProto.Pieces.Count ||\n                modelProto.TrainerSpec.UnkId >= modelProto.Pieces.Count)\n            {\n                throw new ArgumentException(\"The BOS, EOS, or UNK token is not present in the vocabulary.\");\n            }\n\n            _vocabReverse = new (string Piece, float Score, ModelProto.Types.SentencePiece.Types.Type Type)[modelProto.Pieces.Count];\n\n            _minScore = float.MaxValue;\n            _maxScore = float.MinValue;\n\n            for (int i = 0; i < modelProto.Pieces.Count; i++)\n            {\n                if (modelProto.Pieces[i].Type == ModelProto.Types.SentencePiece.Types.Type.Normal ||\n                    modelProto.Pieces[i].Type == ModelProto.Types.SentencePiece.Types.Type.UserDefined ||\n                    modelProto.Pieces[i].Type == ModelProto.Types.SentencePiece.Types.Type.Unused)\n                {\n                    string piece = modelProto.Pieces[i].Piece;\n                    float score = modelProto.Pieces[i].Score;\n                    _vocabReverse[i] = (piece, score, modelProto.Pieces[i].Type);\n                    _vocab.Add(piece, i);\n                    _minScore = Math.Min(_minScore, score);","sourceCodeStart":20,"sourceCodeEnd":56,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceUnigramModel.cs#L20-L56","documentation":"When constructing a SentencePieceUnigramModel from a SentencePiece model proto, the trainer spec's bos_id, eos_id, and unk_id must each index an existing entry in the pieces list. If any of these ids is >= the number of pieces, the special token would reference a nonexistent vocabulary entry, so the constructor throws ArgumentException to prevent a broken model.","triggerScenarios":"Creating SentencePieceUnigramModel from a model proto whose TrainerSpec declares BosId/EosId/UnkId values outside the Pieces range — typically a truncated, partially exported, or corrupted .model/spiece.model file.","commonSituations":"Manually constructed or trimmed SentencePiece protos; files truncated during download or transfer; conversion scripts that dropped pieces but left the original trainer ids; protos built programmatically with placeholder ids.","solutions":["Re-export the SentencePiece model with the official SentencePiece trainer/exporter so pieces and trainer spec ids are consistent.","Re-download the model file and compare checksums/size — a truncated file can lose trailing pieces.","Set the offending id (bos_id/eos_id/unk_id) to -1 in the trainer spec if that token genuinely does not exist in this model.","Inspect the proto (e.g. with Python sentencepiece) to confirm Pieces.Count exceeds the max of bos_id/eos_id/unk_id before loading in .NET."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// C# — sanity-check the proto before construction\nint pieceCount = modelProto.Pieces.Count;\nbool idsValid = modelProto.TrainerSpec.BosId < pieceCount &&\n                modelProto.TrainerSpec.EosId < pieceCount &&\n                modelProto.TrainerSpec.UnkId < pieceCount;","typeGuard":"static bool HasValidSpecialIds(ModelProto p) =>\n    p.TrainerSpec.BosId < p.Pieces.Count &&\n    p.TrainerSpec.EosId < p.Pieces.Count &&\n    p.TrainerSpec.UnkId < p.Pieces.Count;","tryCatchPattern":"try { var model = new SentencePieceUnigramModel(modelProto); }\ncatch (ArgumentException ex) { /* model file corrupt/truncated — re-export or re-download */ }","preventionTips":["Verify model file checksums after download.","Re-export .model files with official SentencePiece tooling.","Never trim Pieces without updating TrainerSpec ids.","Probe with Python sentencepiece (sp.bos_id()/eos_id()/unk_id() < sp.GetPieceSize()) before loading."],"tags":["sentencepiece","model-loading","vocabulary","corrupt-file"],"backgroundTag":"invalid-argument-value","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}