dotnet/machinelearning · error · InvalidDataException

The tokenizer.json post_processor special token '{tokenName}

Error message

The tokenizer.json post_processor special token '{tokenName}' maps to id {id}, which does not match the vocabulary or added tokens.

What it means

After resolving a template special token's numeric id, the library verifies consistency: the id must equal the id recorded in special_tokens, or vocab[id].Piece must equal the token text. An id that matches neither means decoding would yield a different token than intended, so InvalidDataException is thrown.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs:941

                st.TryGetProperty(tokenName, out JsonElement entry) &&
                entry.TryGetProperty("ids", out JsonElement ids) &&
                ids.ValueKind == JsonValueKind.Array &&
                ids.GetArrayLength() > 0)
            {
                if (ids[0].ValueKind != JsonValueKind.Number)
                {
                    throw new InvalidDataException($"The tokenizer.json post_processor special token '{tokenName}' has a non-numeric id.");
                }

                int id = ids[0].GetInt32();

                // Validate the id maps back to the referenced token (via added tokens or the vocab), mirroring
                // AddProcessorAffix, so an inconsistent file cannot emit an id whose decoded token differs.
                bool consistent = (specialTokens.TryGetValue(tokenName, out int mappedId) && mappedId == id)
                    || (id >= 0 && id < vocab.Count && vocab[id].Piece == tokenName);
                if (!consistent)
                {
                    throw new InvalidDataException($"The tokenizer.json post_processor special token '{tokenName}' maps to id {id}, which does not match the vocabulary or added tokens.");
                }

                return id;
            }

            if (specialTokens.TryGetValue(tokenName, out int specialId))
            {
                return specialId;
            }

            int vocabId = FindPieceId(vocab, tokenName);
            if (vocabId < 0)
            {
                throw new InvalidDataException($"The tokenizer.json post_processor references special token '{tokenName}' that is not present in the vocabulary.");
            }

            return vocabId;
        }

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Ensure special_tokens ids match either the added_tokens mapping or vocab[id].Piece == tokenName; regenerate the file from a single consistent tokenizer
2. Load vocab, added_tokens, and post_processor from the same model version — do not mix revisions
3. Re-export tokenizer.json with save_pretrained after any vocabulary change
4. Catch InvalidDataException and fall back to rebuilding affixes from added_tokens/vocab directly

Example fix

// before (vocab has "</s>" at 2 but special_tokens says ids: [5])
"</s>": {"ids": [5]}
// after
"</s>": {"ids": [2]}

int id = specialTokens[token].ids[0];
bool ok = (addedTokens.TryGetValue(token, out var m) && m == id) ||
          (id >= 0 && id < vocab.Count && vocab[id] == token);
if (!ok) throw new InvalidDataException("Inconsistent special token id");

static bool IsConsistentSpecialToken(string token, int id, IReadOnlyDictionary<string,int> added, IReadOnlyList<string> vocab) =>
    (added.TryGetValue(token, out var m) && m == id) ||
    (id >= 0 && id < vocab.Count && vocab[id] == token);

try { tok = SentencePieceTokenizer.Create(...); }
catch (InvalidDataException ex) when (ex.Message.Contains("does not match the vocabulary"))
{ /* reload all files from the same revision */ }

When it happens

Trigger: tokenizer.json where post_processor special_tokens[tokenName].ids[0] disagrees with both the top-level added_tokens map and the vocabulary (e.g. ids shifted after vocab edits, or a copied special_tokens block from a different tokenizer).

Common situations: Merging a post_processor from one model with the vocab of another; editing the vocabulary after saving; mismatched tokenizer.json/vocab files fetched from different model revisions.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• The post-processor '{property}' token '{token}' with id {id}
• An 'added_tokens' entry must have a string 'content' and a n
• tokenizer.json post_processor templates with more than one s
• A post_processor template 'SpecialToken.id' must be a string
• tokenizer.json post_processor template does not contain a se

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/027757019052c729. Report an issue: GitHub.