dotnet/machinelearning · critical · ArgumentException

The BOS, EOS, or UNK token is not present in the vocabulary.

Error message

The BOS, EOS, or UNK token is not present in the vocabulary.

What it means

When constructing a SentencePieceUnigramModel from a SentencePiece model proto, the trainer spec's bos_id, eos_id, and unk_id must each index an existing entry in the pieces list. If any of these ids is >= the number of pieces, the special token would reference a nonexistent vocabulary entry, so the constructor throws ArgumentException to prevent a broken model.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceUnigramModel.cs:38

    {
        private readonly SortedDictionary<string, int> _vocab;
        private readonly (string Piece, float Score, ModelProto.Types.SentencePiece.Types.Type Type)[] _vocabReverse;
        private readonly DoubleArrayTrie _trie;
        private readonly float _minScore;
        private readonly float _maxScore;
        private readonly (int Id, string Token)[] _prefixTokens;
        private readonly (int Id, string Token)[] _suffixTokens;
        private const float UnkPenalty = 10.0f;

        public SentencePieceUnigramModel(ModelProto modelProto, bool addBos, bool addEos, IReadOnlyDictionary<string, int>? specialTokens = null) : base(modelProto, addBos, addEos, specialTokens)
        {
            _vocab = new SortedDictionary<string, int>(OrdinalUtf8StringComparer.Instance);

            if (modelProto.TrainerSpec.BosId >= modelProto.Pieces.Count ||
                modelProto.TrainerSpec.EosId >= modelProto.Pieces.Count ||
                modelProto.TrainerSpec.UnkId >= modelProto.Pieces.Count)
            {
                throw new ArgumentException("The BOS, EOS, or UNK token is not present in the vocabulary.");
            }

            _vocabReverse = new (string Piece, float Score, ModelProto.Types.SentencePiece.Types.Type Type)[modelProto.Pieces.Count];

            _minScore = float.MaxValue;
            _maxScore = float.MinValue;

            for (int i = 0; i < modelProto.Pieces.Count; i++)
            {
                if (modelProto.Pieces[i].Type == ModelProto.Types.SentencePiece.Types.Type.Normal ||
                    modelProto.Pieces[i].Type == ModelProto.Types.SentencePiece.Types.Type.UserDefined ||
                    modelProto.Pieces[i].Type == ModelProto.Types.SentencePiece.Types.Type.Unused)
                {
                    string piece = modelProto.Pieces[i].Piece;
                    float score = modelProto.Pieces[i].Score;
                    _vocabReverse[i] = (piece, score, modelProto.Pieces[i].Type);
                    _vocab.Add(piece, i);
                    _minScore = Math.Min(_minScore, score);

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Re-export the SentencePiece model with the official SentencePiece trainer/exporter so pieces and trainer spec ids are consistent.
2. Re-download the model file and compare checksums/size — a truncated file can lose trailing pieces.
3. Set the offending id (bos_id/eos_id/unk_id) to -1 in the trainer spec if that token genuinely does not exist in this model.
4. Inspect the proto (e.g. with Python sentencepiece) to confirm Pieces.Count exceeds the max of bos_id/eos_id/unk_id before loading in .NET.

// C# — sanity-check the proto before construction
int pieceCount = modelProto.Pieces.Count;
bool idsValid = modelProto.TrainerSpec.BosId < pieceCount &&
                modelProto.TrainerSpec.EosId < pieceCount &&
                modelProto.TrainerSpec.UnkId < pieceCount;

static bool HasValidSpecialIds(ModelProto p) =>
    p.TrainerSpec.BosId < p.Pieces.Count &&
    p.TrainerSpec.EosId < p.Pieces.Count &&
    p.TrainerSpec.UnkId < p.Pieces.Count;

try { var model = new SentencePieceUnigramModel(modelProto); }
catch (ArgumentException ex) { /* model file corrupt/truncated — re-export or re-download */ }

When it happens

Trigger: Creating SentencePieceUnigramModel from a model proto whose TrainerSpec declares BosId/EosId/UnkId values outside the Pieces range — typically a truncated, partially exported, or corrupted .model/spiece.model file.

Common situations: Manually constructed or trimmed SentencePiece protos; files truncated during download or transfer; conversion scripts that dropped pieces but left the original trainer ids; protos built programmatically with placeholder ids.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• The tokenizer.json model enables byte_fallback but does not
• The vocabulary does not contain the required special token.
• ArgumentNullException
• vocab
• unkId must be a valid index in the vocabulary.

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/c8c3f601260fdcb9. Report an issue: GitHub.