dotnet/machinelearning · error · InvalidDataException

The tokenizer.json post_processor references special token

Error message

The tokenizer.json post_processor references special token '{tokenName}' that is not present in the vocabulary.

What it means

If a template references a special token name that appears in neither the post_processor special_tokens map, the added_tokens, nor the vocabulary, FindPieceId returns -1 and InvalidDataException is thrown. The tokenizer refuses to emit ids for tokens it cannot resolve.

Solutions

Correct the token name in the template to one present in the vocabulary or added_tokens (e.g. "<s>" instead of "[CLS]")
Add the missing token to added_tokens with a valid id, or add it to the vocab
Use a post_processor that belongs to the same tokenizer/model as the vocabulary
Pre-check each template token with a lookup over added_tokens and vocab before loading

Example fix

// before (SentencePiece vocab has no [CLS])
{"SpecialToken": {"id": "[CLS]", "ids": [101]}}
// after
{"SpecialToken": {"id": "<s>", "ids": [1]}}

Defensive patterns

Strategy: validation

Validate before calling

bool present = addedTokens.ContainsKey(name) || vocab.Contains(name);
if (!present) throw new KeyNotFoundException($"Special token '{name}' not in vocab");

Type guard

static bool TokenExists(string name, IReadOnlyDictionary<string,int> added, IReadOnlyList<string> vocab) =>
    added.ContainsKey(name) || vocab.Contains(name);

Try / catch

try { tok = SentencePieceTokenizer.Create(...); }
catch (InvalidDataException ex) when (ex.Message.Contains("not present in the vocabulary"))
{ /* add the token or use the matching post_processor */ }

Prevention

Match the post_processor to the tokenizer it shipped with
Check for SentencePiece (<s>) vs BERT ([CLS]) token naming differences
Spell-check template token names against the vocab

When it happens

Trigger: tokenizer.json post_processor template SpecialToken whose id string is not found via specialTokens or FindPieceId over the SentencePiece vocab — e.g. referencing "[CLS]" in a SentencePiece model that only defines "<s>", or a typo like "<unk>" vs "unk".

Common situations: Copying a post_processor (with BERT-style tokens) onto a SentencePiece tokenizer; renamed special tokens between tokenizer versions; typos in hand-edited templates.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/46803ff36581a88f. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs:955

                bool consistent = (specialTokens.TryGetValue(tokenName, out int mappedId) && mappedId == id)
                    || (id >= 0 && id < vocab.Count && vocab[id].Piece == tokenName);
                if (!consistent)
                {
                    throw new InvalidDataException($"The tokenizer.json post_processor special token '{tokenName}' maps to id {id}, which does not match the vocabulary or added tokens.");
                }

                return id;
            }

            if (specialTokens.TryGetValue(tokenName, out int specialId))
            {
                return specialId;
            }

            int vocabId = FindPieceId(vocab, tokenName);
            if (vocabId < 0)
            {
                throw new InvalidDataException($"The tokenizer.json post_processor references special token '{tokenName}' that is not present in the vocabulary.");
            }

            return vocabId;
        }

        private static void AddProcessorAffix(
            JsonElement postProcessor,
            string property,
            List<(int Id, string Token)> target,
            IReadOnlyList<(string Piece, float Score)> vocab,
            IReadOnlyDictionary<string, int> specialTokens)
        {
            // Roberta/Bert processors store cls/sep as [token, id] arrays.
            if (postProcessor.TryGetProperty(property, out JsonElement el) && el.ValueKind == JsonValueKind.Array && el.GetArrayLength() >= 2 &&
                el[0].ValueKind == JsonValueKind.String && el[1].ValueKind == JsonValueKind.Number)
            {
                string token = el[0].GetString()!;
                int id = el[1].GetInt32();

View on GitHub (pinned to 7b76e69cf9)