{"record":{"id":"730f9325bada61c6","repo":"PaddlePaddle/PaddleOCR","slug":"found-a-token-class-in-the-saved-added-toke","errorCode":null,"errorMessage":"Found a {token.__class__} in the saved `added_tokens_decoder`, should be a dictionary or an AddedToken instance","messagePattern":"Found a (.+?) in the saved `added_tokens_decoder`, should be a dictionary or an AddedToken instance","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"ppocr/data/imaug/label_ops.py","lineNumber":1990,"sourceCode":"        )\n        self.tokenizer = TokenizerFast.from_file(fast_tokenizer_file)\n        added_tokens_decoder = {}\n        added_tokens_map = {}\n\n        if tokenizer_config_file is not None:\n            with open(\n                tokenizer_config_file, encoding=\"utf-8\"\n            ) as tokenizer_config_handle:\n                init_kwargs = json.load(tokenizer_config_handle)\n                if \"added_tokens_decoder\" in init_kwargs:\n                    for idx, token in init_kwargs[\"added_tokens_decoder\"].items():\n                        if isinstance(token, dict):\n                            token = AddedToken(**token)\n                        if isinstance(token, AddedToken):\n                            added_tokens_decoder[int(idx)] = token\n                            added_tokens_map[str(token)] = token\n                        else:\n                            raise ValueError(\n                                f\"Found a {token.__class__} in the saved `added_tokens_decoder`, should be a dictionary or an AddedToken instance\"\n                            )\n                init_kwargs[\"added_tokens_decoder\"] = added_tokens_decoder\n                added_tokens_decoder = init_kwargs.pop(\"added_tokens_decoder\", {})\n                tokens_to_add = [\n                    token\n                    for index, token in sorted(\n                        added_tokens_decoder.items(), key=lambda x: x[0]\n                    )\n                    if token not in added_tokens_decoder\n                ]\n                added_tokens_encoder = self.added_tokens_encoder(added_tokens_decoder)\n                encoder = list(added_tokens_encoder.keys()) + [\n                    str(token) for token in tokens_to_add\n                ]\n                tokens_to_add += [\n                    token\n                    for token in self.all_special_tokens_extended","sourceCodeStart":1972,"sourceCodeEnd":2008,"githubUrl":"https://github.com/PaddlePaddle/PaddleOCR/blob/2661c7c0ef5c613e8f93c6e93b2e052399f0f854/ppocr/data/imaug/label_ops.py#L1972-L2008","documentation":"When loading a tokenizer, this code reads tokenizer_config.json and rehydrates the added_tokens_decoder mapping: each entry must be a dict (converted to AddedToken via **kwargs) or an AddedToken instance. Any other JSON type (string, list, number) raises ValueError naming the offending class.","triggerScenarios":"Loading a tokenizer whose tokenizer_config.json contains an added_tokens_decoder entry like {\"0\": \"<pad>\"} instead of {\"0\": {\"content\": \"<pad>\", ...}}, typically a hand-edited or third-party-converted config.","commonSituations":"Manually editing tokenizer_config.json to add special tokens; using a tokenizer converted by an older/newer tool that writes a flat string map; mixing tokenizer files from different transformers versions.","solutions":["Inspect tokenizer_config.json: python -c 'import json; d=json.load(open(\"tokenizer_config.json\")); print({k: type(v).__name__ for k,v in d.get(\"added_tokens_decoder\",{}).items()})' and fix any non-dict entry to the AddedToken kwargs form {\"content\": ..., \"lstrip\": false, ...}","Regenerate the config by loading the tokenizer with a matching transformers version and calling tokenizer.save_pretrained(...)","Delete a corrupted added_tokens_decoder block only if no custom added tokens are needed, then re-save","Restore the original tokenizer files from the model repository you downloaded"],"exampleFix":"// before (tokenizer_config.json)\n\"added_tokens_decoder\": { \"0\": \"<pad>\" }\n// after\n\"added_tokens_decoder\": { \"0\": { \"content\": \"<pad>\", \"lstrip\": false, \"rstrip\": false, \"single_word\": false, \"special\": true } }","handlingStrategy":"validation","validationCode":"import json\ncfg = json.load(open('tokenizer_config.json', encoding='utf-8'))\nfor idx, tok in cfg.get('added_tokens_decoder', {}).items():\n    if not isinstance(tok, dict):\n        raise SystemExit(f'added_tokens_decoder[{idx}] must be a dict, got {type(tok).__name__}: {tok!r}')","typeGuard":"def is_valid_added_tokens_decoder(cfg: dict) -> bool:\n    return all(isinstance(v, dict) and 'content' in v\n               for v in cfg.get('added_tokens_decoder', {}).values())","tryCatchPattern":"try:\n    tok = TokenClass.from_pretrained(path)\nexcept ValueError as e:\n    if 'added_tokens_decoder' in str(e):\n        raise RuntimeError(f'Corrupted tokenizer_config.json at {path}; regenerate with save_pretrained') from e\n    raise","preventionTips":["Never hand-edit added_tokens_decoder; load, modify via add_tokens, then save_pretrained","Keep tokenizer directories immutable and re-download instead of patching","Validate the JSON schema of tokenizer configs in a CI step for fine-tuned models"],"tags":["tokenizer","json","config-file","corruption"],"backgroundTag":null,"analyzedSha":"2661c7c0ef5c613e8f93c6e93b2e052399f0f854","analyzedAt":"2026-08-14T20:17:30.180Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}