{"record":{"id":"9670466d193efb46","repo":"dotnet/machinelearning","slug":"the-post-processor-property-token-token-wi","errorCode":null,"errorMessage":"The post-processor '{property}' token '{token}' with id {id} does not match the vocabulary or added tokens.","messagePattern":"The post-processor '(.+?)' token '(.+?)' with id (.+?) does not match the vocabulary or added tokens\\.","errorType":"validation","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs","lineNumber":981,"sourceCode":"            string property,\n            List<(int Id, string Token)> target,\n            IReadOnlyList<(string Piece, float Score)> vocab,\n            IReadOnlyDictionary<string, int> specialTokens)\n        {\n            // Roberta/Bert processors store cls/sep as [token, id] arrays.\n            if (postProcessor.TryGetProperty(property, out JsonElement el) && el.ValueKind == JsonValueKind.Array && el.GetArrayLength() >= 2 &&\n                el[0].ValueKind == JsonValueKind.String && el[1].ValueKind == JsonValueKind.Number)\n            {\n                string token = el[0].GetString()!;\n                int id = el[1].GetInt32();\n\n                // Validate the [token, id] pair against the vocabulary / added tokens so an inconsistent file cannot\n                // emit ids that do not map to the intended token.\n                bool consistent = (specialTokens.TryGetValue(token, out int specialId) && specialId == id)\n                    || (id >= 0 && id < vocab.Count && vocab[id].Piece == token);\n                if (!consistent)\n                {\n                    throw new InvalidDataException($\"The post-processor '{property}' token '{token}' with id {id} does not match the vocabulary or added tokens.\");\n                }\n\n                target.Add((id, token));\n            }\n        }\n\n        private static void AddAffixToken(\n            List<(int Id, string Token)> target,\n            string tokenName,\n            IReadOnlyList<(string Piece, float Score)> vocab,\n            IReadOnlyDictionary<string, int> specialTokens,\n            bool required)\n        {\n            int id = specialTokens.TryGetValue(tokenName, out int specialId) ? specialId : FindPieceId(vocab, tokenName);\n            if (id >= 0)\n            {\n                target.Add((id, tokenName));\n            }","sourceCodeStart":963,"sourceCodeEnd":999,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs#L963-L999","documentation":"AddProcessorAffix validates each [token, id] pair coming from the post-processor's begin/end affix lists (e.g. BertProcessing style fields): the id must equal the id in specialTokens, or vocab[id].Piece must equal the token text. A mismatch means the file would emit ids decoding to the wrong token, so InvalidDataException names the property, token, and id.","triggerScenarios":"tokenizer.json post_processor containing token/id pairs (from properties like 'sep'/'cls' or the Sequence affix lists) whose numeric id disagrees with both added_tokens and the vocabulary — typically after vocab edits, index shifts, or reusing a post_processor from a different model.","commonSituations":"Mixing tokenizer.json sections from different checkpoints; editing the vocab (adding/removing pieces) without updating the post_processor ids; older exported files updated partially by tooling.","solutions":["Update the post_processor token ids so they match added_tokens or vocab positions, or regenerate tokenizer.json wholly via save_pretrained","Load all tokenizer components from the same model revision","Remove the post_processor and construct affixes manually with correct ids from the vocab","Catch InvalidDataException and fall back to a tokenizer built without the affix post-processor"],"exampleFix":"// before (\"</s>\" is id 2 in vocab, post-processor says 5)\n\"post_processor\": {\"sep\": [\"</s>\", 5]}\n// after\n\"post_processor\": {\"sep\": [\"</s>\", 2]}","handlingStrategy":"validation","validationCode":"bool ok = (added.TryGetValue(token, out var m) && m == id) ||\n          (id >= 0 && id < vocab.Count && vocab[id] == token);\nif (!ok) throw new InvalidDataException($\"{property} affix inconsistent\");","typeGuard":"static bool IsValidAffix(string property, string token, int id, IReadOnlyDictionary<string,int> added, IReadOnlyList<string> vocab) =>\n    (added.TryGetValue(token, out var m) && m == id) ||\n    (id >= 0 && id < vocab.Count && vocab[id] == token);","tryCatchPattern":"try { tok = SentencePieceTokenizer.Create(...); }\ncatch (InvalidDataException ex) when (ex.Message.Contains(\"does not match the vocabulary or added tokens\"))\n{ /* re-export tokenizer.json from one consistent source */ }","preventionTips":["Update post_processor ids whenever the vocab changes","Export all tokenizer components together in one save call","Diff tokenizer.json against the hub copy after edits"],"tags":["tokenizer","consistency","post-processor","invalid-data"],"backgroundTag":"invalid-argument-value","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}