dotnet/machinelearning · error · ArgumentException

The special token '{token}' is not in the vocabulary.

Error message

The special token '{token}' is not in the vocabulary.

What it means

The private helper AddSpecialToken throws ArgumentException when the token is null or is not present in the vocabulary. It is invoked from Create while registering options.SpecialTokens, so a token absent from vocab.txt cannot be registered as special. Matching against the vocab is ordinal, so casing/spelling must be exact.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/BertTokenizer.cs:818

                    AddSpecialToken(vocab, tempSpecialTokens, options.MaskingToken, lowerCase, notNormalizedSpecialTokens);

                    options.SpecialTokens = notNormalizedSpecialTokens;
                    specialTokensDict = tempSpecialTokens;
                }
            }

            // We set the PreTokenizer here using the normalized special tokens dict (if relevant), and therefore we can 
            // keep the not-normalized special tokens dict in the options passed to the WordPieceTokenizer.
            options.PreTokenizer ??= options.ApplyBasicTokenization ? PreTokenizer.CreateWordOrPunctuation(options.SplitOnSpecialTokens ? specialTokensDict : null) : PreTokenizer.CreateWhiteSpace();

            return new BertTokenizer(vocab, vocabReverse, options);
        }

        private static void AddSpecialToken(Dictionary<StringSpanOrdinalKey, int> vocab, Dictionary<string, int> specialTokens, string token, bool lowerCase, Dictionary<string, int>? notNormalizedSpecialTokens = null)
        {
            if (token is null || !vocab.TryGetValue(new StringSpanOrdinalKey(token), out int id))
            {
                throw new ArgumentException($"The special token '{token}' is not in the vocabulary.");
            }

            if (notNormalizedSpecialTokens is not null)
            {
                notNormalizedSpecialTokens[token] = id;
            }

            string normalizedToken = token;
            if (lowerCase)
            {
                // Lowercase the special tokens to have the pre-tokenization can find them as we lowercase the input text.
                // we don't even need to do case-insensitive comparisons as we are lowercasing the input text.
                normalizedToken = token.ToLowerInvariant();

                // Add lowercased special tokens to the vocab if they are not already there.
                // This will allow matching during the encoding process.
                vocab[new StringSpanOrdinalKey(normalizedToken)] = id;
            }

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Use only token strings that literally exist in the vocabulary file.
2. Verify exact spelling/case of the special token against vocab.txt (matching is ordinal).
3. If the token is genuinely missing, extend the vocabulary or remove it from SpecialTokens.
4. Ensure no null entries are added to the SpecialTokens dictionary.

Example fix

// before
options.SpecialTokens["[SEPERATOR]"] = 102; // misspelled, not in vocab
// after
options.SpecialTokens["[SEP]"] = 102; // exact vocab entry

// Confirm each special token exists in vocab.txt before construction
bool inVocab = File.ReadLines(vocabPath).Contains(token, StringComparer.Ordinal);

static bool IsKnownSpecialToken(string? token, HashSet<string> vocab) => !string.IsNullOrEmpty(token) && vocab.Contains(token);

try { var t = BertTokenizer.Create(vocabPath, options); } catch (ArgumentException ex) when (ex.Message.Contains("is not in the vocabulary")) { /* drop the unknown special token and retry */ }

When it happens

Trigger: A special token string in BertOptions.SpecialTokens that does not appear in the vocabulary, or a null token entry; reached via BertTokenizer.Create with mismatched options rather than called directly by user code.

Common situations: Vocabulary files missing standard tokens like [CLS]/[SEP]/[MASK] (truncated or custom-built vocabs); token strings with casing/whitespace that don't match vocab entries; IDs corrected to match but the token string itself still misspelled.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• The special token '{kvp.Key}' is not in the vocabulary or as
• The Unknown token '{UnknownToken}' is not found in the vocab
• throw new ArgumentNullException(nameof(vocabStream));
• The beginning of sentence token '{beginningOfSentenceToken}'
• The end of sentence token '{endOfSentenceToken}' was not pre

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/67e133b02ab7b80a. Report an issue: GitHub.