dotnet/machinelearning · error · InvalidOperationException

Unknown Token '{value}' was not present in '{nameof(Vocabula

Error message

Unknown Token '{value}' was not present in '{nameof(Vocabulary)}'.

What it means

The UnknownToken property setter of BpeTokenizer validates that the requested token actually exists in the loaded vocabulary; if the vocab lookup fails, it throws this InvalidOperationException. This fails fast so 'unknown token' handling does not silently map to a nonexistent id.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs:62

        public string? UnknownToken
        {
            get
            {
                return _unknownToken;
            }

            private set
            {
                if (value is null)
                {
                    _unknownToken = value;
                    _unknownTokenId = null;
                    return;
                }

                if (!_vocab.TryGetValue(value, out int id))
                {
                    throw new InvalidOperationException($"Unknown Token '{value}' was not present in '{nameof(Vocabulary)}'.");
                }

                _unknownTokenId = id;
                _unknownToken = value;
            }
        }

        /// <summary>
        /// A prefix to be used for every subword that is not a beginning-of-word
        /// </summary>
        public string? ContinuingSubwordPrefix { get; }

        /// <summary>
        /// An optional suffix to characterize the end-of-word and sub-word
        /// </summary>
        public string? EndOfWordSuffix { get; }

        /// <summary>

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Add the token you want as unknown (e.g. '<unk>') to the vocabulary file/dictionary with a valid id.
2. Use a token that is actually present in the vocabulary as UnknownToken (check the vocab file for the exact spelling).
3. Leave UnknownToken unset (null) if you do not need custom unknown-token handling — the setter returns early for null.
4. Verify with a vocab lookup before constructing the tokenizer.

Example fix

// before
options.UnknownToken = "<unk>"; // not in vocab -> throws
// after
options.Vocabulary["<unk>"] = options.Vocabulary.Count; // add first
options.UnknownToken = "<unk>";

if (unknownToken != null && !vocab.ContainsKey(unknownToken))
    throw new InvalidOperationException($"UnknownToken '{unknownToken}' missing from vocabulary.");

try { options.UnknownToken = tok; tokenizer = BpeTokenizer.Create(options); }
catch (InvalidOperationException ex) when (ex.Message.Contains("was not present in"))
{ options.UnknownToken = null; tokenizer = BpeTokenizer.Create(options); }

When it happens

Trigger: Setting BpeTokenizer.UnknownToken (or BpeOptions.UnknownToken passed to BpeTokenizer.Create) to a string such as '<unk>' or '[UNK]' that is not a key in the supplied Vocabulary.

Common situations: Copying the unknown-token string from another tokenizer family (Hugging Face, SentencePiece) whose vocab uses a different sentinel; loading a trimmed/partial vocab file that dropped the <unk> entry; typos like '<unk>' vs '<UNK>'.

Understand the failure class

Background: "Not found" and "does not exist" errors: why "Task not found", "No such folder", and "Can't find" fire when a lookup comes back empty — this error's family across 14 libraries.

Related errors

• The vocabulary cannot be null.
• The vocabulary cannot be empty.
• The merge entries cannot be null.
• Invalid merger file format
• The beginning of sentence token '{beginningOfSentenceToken}'

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/5adbe77e7f186578. Report an issue: GitHub.