dotnet/machinelearning · error · InvalidOperationException

Unknown Token Out Of Vocabulary.

Error message

Unknown Token Out Of Vocabulary.

What it means

During BPE encoding, when an unknown character sequence is fused and the tokenizer maps the UnknownToken (e.g. '<unk>') to its ID, that token must exist in the loaded vocabulary. If the vocab lacks an entry for UnknownToken, encoding fails with InvalidOperationException 'Unknown Token Out Of Vocabulary.'

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs:1278

                    }
                    word.Add(id, length);
                }
                else if (UnknownToken is not null)
                {
                    if (unk.HasValue)
                    {
                        if (FuseUnknownTokens)
                        {
                            // Fuse unk
                            unk = (unk.Value.Id, unk.Value.Len + length);
                        }
                        else
                        {
                            // Do not fuse unk, add the previous one
                            word.Add(unk.Value.Id, unk.Value.Len);
                            if (!_vocab.TryGetValue(UnknownToken, out int value))
                            {
                                throw new InvalidOperationException($"Unknown Token Out Of Vocabulary.");
                            }
                            unk = (value, length);
                        }
                    }
                    else
                    {
                        if (!_vocab.TryGetValue(UnknownToken, out int value))
                        {
                            throw new InvalidOperationException($"Unknown Token Out Of Vocabulary.");
                        }
                        unk = (value, length);
                    }
                }

                i += length;
            }

            if (unk.HasValue)

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Set UnknownToken to the exact string used in the vocabulary file (commonly '<unk>' or '<|unk|>').
2. Use a vocab that actually contains an unknown-token entry.
3. Prefer byte-level BPE tokenizers, whose byte fallback makes unknown tokens unnecessary.

Example fix

// before
var opts = new BpeOptions(vocab) { UnknownToken = "UNK" }; // vocab has "<unk>", not "UNK"
// after
var opts = new BpeOptions(vocab) { UnknownToken = "<unk>" };

if (!vocab.ContainsKey(unknownToken)) throw new InvalidOperationException($"Vocab is missing unknown token '{unknownToken}'");

try { ids = tokenizer.EncodeToIds(text); }
catch (InvalidOperationException ex) when (ex.Message.Contains("Unknown Token Out Of Vocabulary")) { /* fix vocab/options */ }

When it happens

Trigger: Encoding text containing characters absent from the vocabulary while the configured UnknownToken string has no entry in vocab.json — a mismatch between the UnknownToken setting and the actual vocab contents.

Common situations: Constructing BpeTokenizer with a custom unknownToken value that differs from the vocab's key (case/spacing mismatch), or using a vocab file with no <unk> entry and byte-level encoding disabled.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Unknown Token '{value}' was not present in '{nameof(Vocabula
• The vocabulary cannot be null.
• The max token count must be greater than 0.
• Invalid merger file format at line: {lineNumber}
• The special token '{kvp.Key}' is not in the vocabulary or as

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/df4f4864a74444c1. Report an issue: GitHub.