dotnet/machinelearning · error · InvalidDataException

The tokenizer.json does not contain the required special…

Error message

The tokenizer.json does not contain the required special token '{tokenName}'.

What it means

AddAffixToken looks up a required post-processor affix token (e.g. BOS/EOS for the Sequence template) in the vocabulary/added tokens; if the id is not found (-1) and the token was marked required, InvalidDataException is thrown because the processor cannot build its prefix/suffix.

Solutions

Add the missing special token to tokenizer.json's added_tokens (and model vocab) with a valid id, or re-export the tokenizer
Change the post_processor template to reference tokens that exist in this vocabulary
Set the affix as optional / remove it from the template if the token is genuinely not needed
Check that the downloaded tokenizer.json matches the model card (unstripped version)

Example fix

// before (added_tokens lacks "</s>")
{"added_tokens": [{"content": "<s>", "id": 1}]}
// after
{"added_tokens": [{"content": "<s>", "id": 1}, {"content": "</s>", "id": 2}]}

Defensive patterns

Strategy: validation

Validate before calling

int id = addedTokens.TryGetValue(name, out var a) ? a
        : vocab.IndexOf(name);
if (id < 0 && required) throw new InvalidOperationException($"Missing required token '{name}'");

Type guard

static bool RequiredTokenPresent(string name, IReadOnlyDictionary<string,int> added, IReadOnlyList<string> vocab) =>
    added.ContainsKey(name) || vocab.Contains(name);

Try / catch

try { tok = SentencePieceTokenizer.Create(...); }
catch (InvalidDataException ex) when (ex.Message.Contains("required special token"))
{ /* use the unstripped tokenizer.json or add the token */ }

Prevention

Verify required BOS/EOS tokens exist before loading pruned tokenizers
Download tokenizer.json from the model card link, not third-party copies
List added_tokens during onboarding of a new tokenizer

When it happens

Trigger: ResolvePostProcessorAffixes requests a mandatory special token (for example "<s>" or "</s>") that is absent from both the tokenizer.json added_tokens and the vocabulary — e.g. a Metaspace post_processor template referencing EOS in a model that never defined it.

Common situations: Stripped or pruned tokenizers with special tokens removed; using a base-model tokenizer.json where the post-processor expects tokens only present in fine-tuned variants; typos in template token names.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/633c06a626a2cee3. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs:1002

                target.Add((id, token));
            }
        }

        private static void AddAffixToken(
            List<(int Id, string Token)> target,
            string tokenName,
            IReadOnlyList<(string Piece, float Score)> vocab,
            IReadOnlyDictionary<string, int> specialTokens,
            bool required)
        {
            int id = specialTokens.TryGetValue(tokenName, out int specialId) ? specialId : FindPieceId(vocab, tokenName);
            if (id >= 0)
            {
                target.Add((id, tokenName));
            }
            else if (required)
            {
                throw new InvalidDataException($"The tokenizer.json does not contain the required special token '{tokenName}'.");
            }
        }

        private static int FindPieceId(IReadOnlyList<(string Piece, float Score)> vocab, string token)
        {
            for (int i = 0; i < vocab.Count; i++)
            {
                if (vocab[i].Piece == token)
                {
                    return i;
                }
            }

            return -1;
        }

        // Reads a string-valued property, returning null when the property is absent or not a JSON string. Keeps the
        // switch/compare parsing paths from throwing InvalidOperationException on malformed (non-string) tokenizer.json.

View on GitHub (pinned to 7b76e69cf9)