dotnet/machinelearning · error · InvalidDataException

The post-processor '{property}' token '{token}' with id {id}

Error message

The post-processor '{property}' token '{token}' with id {id} does not match the vocabulary or added tokens.

What it means

AddProcessorAffix validates each [token, id] pair coming from the post-processor's begin/end affix lists (e.g. BertProcessing style fields): the id must equal the id in specialTokens, or vocab[id].Piece must equal the token text. A mismatch means the file would emit ids decoding to the wrong token, so InvalidDataException names the property, token, and id.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs:981

            string property,
            List<(int Id, string Token)> target,
            IReadOnlyList<(string Piece, float Score)> vocab,
            IReadOnlyDictionary<string, int> specialTokens)
        {
            // Roberta/Bert processors store cls/sep as [token, id] arrays.
            if (postProcessor.TryGetProperty(property, out JsonElement el) && el.ValueKind == JsonValueKind.Array && el.GetArrayLength() >= 2 &&
                el[0].ValueKind == JsonValueKind.String && el[1].ValueKind == JsonValueKind.Number)
            {
                string token = el[0].GetString()!;
                int id = el[1].GetInt32();

                // Validate the [token, id] pair against the vocabulary / added tokens so an inconsistent file cannot
                // emit ids that do not map to the intended token.
                bool consistent = (specialTokens.TryGetValue(token, out int specialId) && specialId == id)
                    || (id >= 0 && id < vocab.Count && vocab[id].Piece == token);
                if (!consistent)
                {
                    throw new InvalidDataException($"The post-processor '{property}' token '{token}' with id {id} does not match the vocabulary or added tokens.");
                }

                target.Add((id, token));
            }
        }

        private static void AddAffixToken(
            List<(int Id, string Token)> target,
            string tokenName,
            IReadOnlyList<(string Piece, float Score)> vocab,
            IReadOnlyDictionary<string, int> specialTokens,
            bool required)
        {
            int id = specialTokens.TryGetValue(tokenName, out int specialId) ? specialId : FindPieceId(vocab, tokenName);
            if (id >= 0)
            {
                target.Add((id, tokenName));
            }

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Update the post_processor token ids so they match added_tokens or vocab positions, or regenerate tokenizer.json wholly via save_pretrained
2. Load all tokenizer components from the same model revision
3. Remove the post_processor and construct affixes manually with correct ids from the vocab
4. Catch InvalidDataException and fall back to a tokenizer built without the affix post-processor

Example fix

// before ("</s>" is id 2 in vocab, post-processor says 5)
"post_processor": {"sep": ["</s>", 5]}
// after
"post_processor": {"sep": ["</s>", 2]}

bool ok = (added.TryGetValue(token, out var m) && m == id) ||
          (id >= 0 && id < vocab.Count && vocab[id] == token);
if (!ok) throw new InvalidDataException($"{property} affix inconsistent");

static bool IsValidAffix(string property, string token, int id, IReadOnlyDictionary<string,int> added, IReadOnlyList<string> vocab) =>
    (added.TryGetValue(token, out var m) && m == id) ||
    (id >= 0 && id < vocab.Count && vocab[id] == token);

try { tok = SentencePieceTokenizer.Create(...); }
catch (InvalidDataException ex) when (ex.Message.Contains("does not match the vocabulary or added tokens"))
{ /* re-export tokenizer.json from one consistent source */ }

When it happens

Trigger: tokenizer.json post_processor containing token/id pairs (from properties like 'sep'/'cls' or the Sequence affix lists) whose numeric id disagrees with both added_tokens and the vocabulary — typically after vocab edits, index shifts, or reusing a post_processor from a different model.

Common situations: Mixing tokenizer.json sections from different checkpoints; editing the vocab (adding/removing pieces) without updating the post_processor ids; older exported files updated partially by tooling.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• The tokenizer.json post_processor special token '{tokenName}
• An 'added_tokens' entry must have a string 'content' and a n
• tokenizer.json post_processor templates with more than one s
• A post_processor template 'SpecialToken.id' must be a string
• tokenizer.json post_processor template does not contain a se

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/9670466d193efb46. Report an issue: GitHub.