dotnet/machinelearning · error · ArgumentException

The vocabulary does not contain the required special token.

Error message

The vocabulary does not contain the required special token.

What it means

CheckSpecialId validates special-token ids (bos, eos, etc.): when the token is marked required and its id is -1 (meaning 'not present'), it throws ArgumentException with the message 'The vocabulary does not contain the required special token.' The library requires that a mandatory special token actually exist in the vocabulary.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceUnigramModel.cs:366

                return -1;
            }

            for (int i = 0; i < pieces.Count; i++)
            {
                if (pieces[i].Piece == tokenName)
                {
                    return i;
                }
            }

            return -1;
        }

        private static int CheckSpecialId(bool required, int id, string paramName)
        {
            if (required && id < 0)
            {
                throw new ArgumentException($"The vocabulary does not contain the required special token.", paramName);
            }
            return id;
        }

        public override IReadOnlyDictionary<string, int> Vocabulary => new ReadOnlyDictionary<string, int>(_vocab);

        public int MaxIdByteFallbackId { get; }

        public override IReadOnlyList<EncodedToken> EncodeToTokens(string? text, ReadOnlySpan<char> textSpan, out string? normalizedText, bool addBeginningOfSentence, bool addEndOfSentence, bool considerNormalization)
        {
            ReadOnlySpan<char> textToEncode = string.IsNullOrEmpty(text) ? textSpan : text.AsSpan();
            if (textToEncode.IsEmpty)
            {
                normalizedText = string.Empty;
                return Array.Empty<EncodedToken>();
            }

            List<EncodedToken> tokens = new();

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Provide a valid id for the required special token in the model configuration (tokenizer.json bos_token/eos_token or proto TrainerSpec).
2. If the model truly has no such token, configure it as optional instead of required so -1 is accepted.
3. Retrain or re-export the SentencePiece model with the needed special tokens defined.
4. Pre-check with Python sentencepiece: confirm sp.bos_id()/sp.eos_id() are not -1 before loading in .NET.

// C# — ensure required special tokens exist before loading
foreach (var (name, id) in new[] { ("bos", bosId), ("eos", eosId) })
{
    if (requireSpecialTokens && id < 0)
        throw new ArgumentException($"Required special token '{name}' not present in vocabulary.");
}

static bool SpecialTokenPresent(bool required, int id) => !required || id >= 0;

try { var model = new SentencePieceUnigramModel(modelProto); }
catch (ArgumentException ex) when (ex.Message.Contains("special token")) { /* model lacks required bos/eos — retrain or mark optional */ }

When it happens

Trigger: Constructing SentencePieceUnigramModel where a required special token (e.g. bos_id or eos_id per the configuration) is -1/absent in the model's trainer spec or id lookup.

Common situations: Models trained without BOS/EOS tokens but loaded with options marking them required; tokenizer.json configs omitting bos_token/eos_token; manually stripped special tokens from the vocabulary.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• The BOS, EOS, or UNK token is not present in the vocabulary.
• The tokenizer.json model enables byte_fallback but does not
• ArgumentNullException
• vocab
• unkId must be a valid index in the vocabulary.

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/5fc53119561eb811. Report an issue: GitHub.