dotnet/machinelearning · error · ArgumentException

The special token '{kvp.Key}' is not in the vocabulary or as

Error message

The special token '{kvp.Key}' is not in the vocabulary or assigned id value {id} different than the value {kvp.Value} in the special tokens.

What it means

During BertTokenizer construction, every entry in BertOptions.SpecialTokens must already exist in the vocabulary with the exact same ID. If a special token string is not found in the vocab, or its vocab ID differs from the ID given in the options, Create throws ArgumentException. This keeps the tokenizer's internal special-token bookkeeping consistent with the vocabulary.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/BertTokenizer.cs:780

            options.Normalizer ??= options.ApplyBasicTokenization ? new BertNormalizer(options.LowerCaseBeforeTokenization, options.IndividuallyTokenizeCjk, options.RemoveNonSpacingMarks) : null;

            IReadOnlyDictionary<string, int>? specialTokensDict = options.SpecialTokens;
            if (options.SplitOnSpecialTokens)
            {
                bool lowerCase = options.ApplyBasicTokenization && options.LowerCaseBeforeTokenization;
                if (options.SpecialTokens is not null)
                {
                    if (lowerCase)
                    {
                        Dictionary<string, int> tempSpecialTokens = [];
                        specialTokensDict = tempSpecialTokens;

                        foreach (var kvp in options.SpecialTokens)
                        {
                            if (!vocab.TryGetValue(new StringSpanOrdinalKey(kvp.Key), out int id) || id != kvp.Value)
                            {
                                throw new ArgumentException($"The special token '{kvp.Key}' is not in the vocabulary or assigned id value {id} different than the value {kvp.Value} in the special tokens.");
                            }

                            // Add the special token into our dictionary, normalizing it, and adding it into the
                            // main vocab, if needed. 
                            AddSpecialToken(vocab, tempSpecialTokens, kvp.Key, true);
                        }
                    }
                }
                else
                {
                    // Create a dictionary with the special tokens - store the un-normalized forms in the options as
                    // that field is exposed to the public. In addition, store the normalized form for creating the 
                    // pre-tokenizer.
                    Dictionary<string, int> tempSpecialTokens = [];
                    Dictionary<string, int> notNormalizedSpecialTokens = [];
                    AddSpecialToken(vocab, tempSpecialTokens, options.UnknownToken, lowerCase, notNormalizedSpecialTokens);
                    AddSpecialToken(vocab, tempSpecialTokens, options.SeparatorToken, lowerCase, notNormalizedSpecialTokens);
                    AddSpecialToken(vocab, tempSpecialTokens, options.PaddingToken, lowerCase, notNormalizedSpecialTokens);

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Remove or correct the mismatched entry in BertOptions.SpecialTokens so token and ID match the vocab file.
2. Open the vocab file and confirm the token's actual ID, then use that ID in options.
3. If the token genuinely doesn't exist, drop it from SpecialTokens or use a vocab that contains it.
4. Load special tokens programmatically from the same source as the vocab rather than hardcoding.

Example fix

// before
var options = new BertOptions { SpecialTokens = { ["[CLS]"] = 99 } }; // wrong id
var tokenizer = BertTokenizer.Create(vocabPath, options);
// after
var options = new BertOptions { SpecialTokens = { ["[CLS]"] = 101 } }; // id as assigned in vocab.txt
var tokenizer = BertTokenizer.Create(vocabPath, options);

// Verify special tokens against the vocab before creating options
foreach (var kvp in options.SpecialTokens)
{
    if (!vocabLookup.TryGetValue(kvp.Key, out var id) || id != kvp.Value)
        throw new InvalidOperationException($"Special token '{kvp.Key}' (id {kvp.Value}) does not match vocab");
}

try { var t = BertTokenizer.Create(vocabPath, options); } catch (ArgumentException ex) when (ex.Message.Contains("special token")) { /* retry with corrected/empty SpecialTokens */ }

When it happens

Trigger: Passing BertOptions.SpecialTokens containing a token string absent from the vocab file; or mapping a token to a different integer ID than the vocab assigns (e.g. copying IDs from a different model's config).

Common situations: Hand-copying special token maps (CLS=101, SEP=102, etc.) from another BERT variant whose vocab IDs differ; typos in special token strings; mixing a vocab.txt from one model with special tokens defined for another.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• The special token '{token}' is not in the vocabulary.
• The Unknown token '{UnknownToken}' is not found in the vocab
• throw new ArgumentNullException(nameof(vocabStream));
• The beginning of sentence token '{beginningOfSentenceToken}'
• The end of sentence token '{endOfSentenceToken}' was not pre

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/d4dff125415829b7. Report an issue: GitHub.