{"record":{"id":"e72aec0c64b15424","repo":"opendatalab/MinerU","slug":"found-a-token-class-in-the-saved-added-toke","errorCode":null,"errorMessage":"Found a {token.__class__} in the saved `added_tokens_decoder`, should be a dictionary or an AddedToken instance","messagePattern":"Found a (.+?) in the saved `added_tokens_decoder`, should be a dictionary or an AddedToken instance","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"mineru/model/mfr/pp_formulanet_plus_m/processors.py","lineNumber":361,"sourceCode":"        self.tokenizer = TokenizerFast.from_buffer(fast_tokenizer_buffer)\n        tokenizer_config = (\n            character_list[\"tokenizer_config_file\"]\n            if \"tokenizer_config_file\" in character_list\n            else None\n        )\n        added_tokens_decoder = {}\n        added_tokens_map = {}\n        if tokenizer_config is not None:\n            init_kwargs = tokenizer_config\n            if \"added_tokens_decoder\" in init_kwargs:\n                for idx, token in init_kwargs[\"added_tokens_decoder\"].items():\n                    if isinstance(token, dict):\n                        token = AddedToken(**token)\n                    if isinstance(token, AddedToken):\n                        added_tokens_decoder[int(idx)] = token\n                        added_tokens_map[str(token)] = token\n                    else:\n                        raise ValueError(\n                            f\"Found a {token.__class__} in the saved `added_tokens_decoder`, should be a dictionary or an AddedToken instance\"\n                        )\n            init_kwargs[\"added_tokens_decoder\"] = added_tokens_decoder\n            added_tokens_decoder = init_kwargs.pop(\"added_tokens_decoder\", {})\n            tokens_to_add = [\n                token\n                for index, token in sorted(\n                    added_tokens_decoder.items(), key=lambda x: x[0]\n                )\n                if token not in added_tokens_decoder\n            ]\n            added_tokens_encoder = self.added_tokens_encoder(added_tokens_decoder)\n            encoder = list(added_tokens_encoder.keys()) + [\n                str(token) for token in tokens_to_add\n            ]\n            tokens_to_add += [\n                token\n                for token in self.all_special_tokens_extended","sourceCodeStart":343,"sourceCodeEnd":379,"githubUrl":"https://github.com/opendatalab/MinerU/blob/4fe4bde114a23ee5dd637eae99b767f4669bf58c/mineru/model/mfr/pp_formulanet_plus_m/processors.py#L343-L379","documentation":"Thrown while loading the tokenizer config for PP-FormulaNet-Plus: each entry of the saved added_tokens_decoder must deserialize into either a dict (which is then converted via AddedToken(**token)) or an AddedToken instance. Any other Python type (str, int, list, None) means the persisted tokenizer_config.json is malformed or was written by an incompatible tokenizers/transformers version.","triggerScenarios":"Loading a tokenizer whose tokenizer_config.json has added_tokens_decoder entries like \"0\": \"<pad>\" (a bare string) instead of \"0\": {\"content\": \"<pad>\", ...}. Also triggered by hand-edited config files or configs generated by a newer transformers version with a changed serialization format.","commonSituations":"Mixing transformers/hf_tokenizers versions between save and load, manually patching tokenizer configs to add special tokens, or truncated/corrupted JSON after a partial download of the model directory.","solutions":["Inspect tokenizer_config.json's added_tokens_decoder and rewrite each entry as {\"content\": ..., \"lstrip\": ..., \"rstrip\": ..., \"single_word\": ...}.","Re-download the model directory in case the config file is truncated or corrupted.","Re-save the tokenizer with the same transformers version used at inference: tokenizer.save_pretrained(dir).","Pin transformers/tokenizers to the version the model card for PP-FormulaNet-Plus recommends."],"exampleFix":"// tokenizer_config.json (before)\n\"added_tokens_decoder\": { \"0\": \"<pad>\" }\n\n// after\n\"added_tokens_decoder\": {\n  \"0\": {\"content\": \"<pad>\", \"lstrip\": false, \"rstrip\": false, \"single_word\": false, \"special\": true}\n}","handlingStrategy":"validation","validationCode":"import json\ndef validate_added_tokens(path):\n    cfg = json.load(open(path))\n    for idx, tok in (cfg.get('added_tokens_decoder') or {}).items():\n        assert isinstance(tok, dict), f'entry {idx} is {type(tok).__name__}, expected dict'\n        assert 'content' in tok, f'entry {idx} missing content'","typeGuard":"def is_valid_token_entry(tok) -> bool:\n    from tokenizers import AddedToken\n    return isinstance(tok, (dict, AddedToken))","tryCatchPattern":"try:\n    model = AutoTokenizer.from_pretrained(model_dir)\nexcept ValueError as e:\n    if 'added_tokens_decoder' in str(e):\n        raise RuntimeError(f'tokenizer_config.json in {model_dir} is malformed; re-download or re-save the model') from e\n    raise","preventionTips":["Pin transformers/tokenizers versions in deployment images to match the model export.","Never hand-edit added_tokens_decoder; use tokenizer.add_tokens() and save_pretrained().","Verify model checksums after download."],"tags":["tokenizer","configuration","model-loading","version-compat"],"backgroundTag":null,"analyzedSha":"4fe4bde114a23ee5dd637eae99b767f4669bf58c","analyzedAt":"2026-08-14T21:29:18.456Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}