{"record":{"id":"424906b42ada6901","repo":"PaddlePaddle/PaddleOCR","slug":"found-a-token-class-in-the-saved-added-toke-424906","errorCode":null,"errorMessage":"Found a {token.__class__} in the saved `added_tokens_decoder`, should be a dictionary or an AddedToken instance","messagePattern":"Found a (.+?) in the saved `added_tokens_decoder`, should be a dictionary or an AddedToken instance","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"ppocr/postprocess/rec_postprocess.py","lineNumber":1344,"sourceCode":"            rec_char_dict_path, \"tokenizer_config.json\"\n        )\n        self.tokenizer = TokenizerFast.from_file(fast_tokenizer_file)\n        added_tokens_decoder = {}\n        added_tokens_map = {}\n        if tokenizer_config_file is not None:\n            with open(\n                tokenizer_config_file, encoding=\"utf-8\"\n            ) as tokenizer_config_handle:\n                init_kwargs = json.load(tokenizer_config_handle)\n                if \"added_tokens_decoder\" in init_kwargs:\n                    for idx, token in init_kwargs[\"added_tokens_decoder\"].items():\n                        if isinstance(token, dict):\n                            token = AddedToken(**token)\n                        if isinstance(token, AddedToken):\n                            added_tokens_decoder[int(idx)] = token\n                            added_tokens_map[str(token)] = token\n                        else:\n                            raise ValueError(\n                                f\"Found a {token.__class__} in the saved `added_tokens_decoder`, should be a dictionary or an AddedToken instance\"\n                            )\n                init_kwargs[\"added_tokens_decoder\"] = added_tokens_decoder\n                added_tokens_decoder = init_kwargs.pop(\"added_tokens_decoder\", {})\n                tokens_to_add = [\n                    token\n                    for index, token in sorted(\n                        added_tokens_decoder.items(), key=lambda x: x[0]\n                    )\n                    if token not in added_tokens_decoder\n                ]\n                added_tokens_encoder = self.added_tokens_encoder(added_tokens_decoder)\n                encoder = list(added_tokens_encoder.keys()) + [\n                    str(token) for token in tokens_to_add\n                ]\n                tokens_to_add += [\n                    token\n                    for token in self.all_special_tokens_extended","sourceCodeStart":1326,"sourceCodeEnd":1362,"githubUrl":"https://github.com/PaddlePaddle/PaddleOCR/blob/2661c7c0ef5c613e8f93c6e93b2e052399f0f854/ppocr/postprocess/rec_postprocess.py#L1326-L1362","documentation":"ValueError from the vendored transformers-style tokenizer loading in rec_postprocess.py: while replaying tokenizer_config.json's added_tokens_decoder, an entry is neither a dict nor an AddedToken instance. The saved file is malformed or was produced/consumed by a library with an incompatible added_tokens_decoder schema.","triggerScenarios":"Initializing the recognition postprocess (e.g. for LaTeX/VLrec-type models like UniMERNet or GenRec tokenizer-based decoding) with a tokenizer_config.json whose added_tokens_decoder maps indices to plain strings, nulls, or serialized non-dict objects.","commonSituations":"Mixing tokenizer file versions: config generated by a newer/older transformers release than the vendored reader expects; hand-merged tokenizer directories; partial downloads truncating the JSON into odd structures; converting checkpoints with third-party scripts that rewrite the file.","solutions":["Regenerate or re-download the matching tokenizer_config.json from the official model release instead of hand-editing it.","Inspect added_tokens_decoder in the file: each value must be an object like {\"content\": \"<unk>\", \"lstrip\": false, ...} or an AddedToken-equivalent dict.","Pin the model repo/converter version that produced the tokenizer so the schema matches the vendored loader.","If a value is a plain string, wrap it: {\"content\": \"<your_token>\", \"single_word\": false, \"lstrip\": false, \"rstrip\": false, \"normalized\": false}."],"exampleFix":"// tokenizer_config.json before\n\"added_tokens_decoder\": {\"0\": \"<pad>\"}\n\n// after\n\"added_tokens_decoder\": {\"0\": {\"content\": \"<pad>\", \"lstrip\": false, \"rstrip\": false, \"single_word\": false, \"normalized\": false}}","handlingStrategy":"validation","validationCode":"import json\n\ndef valid_tokenizer_config(path) -> bool:\n    with open(path, encoding='utf-8') as f:\n        cfg = json.load(f)\n    atd = cfg.get('added_tokens_decoder', {})\n    return all(isinstance(v, dict) for v in atd.values())","typeGuard":"def is_valid_added_tokens_decoder(obj) -> bool:\n    return (\n        isinstance(obj, dict)\n        and all(str(k).isdigit() for k in obj)\n        and all(isinstance(v, dict) for v in obj.values())\n    )","tryCatchPattern":null,"preventionTips":["Take tokenizer files verbatim from the official model release; never hand-merge them.","After converting checkpoints, run a smoke test that loads the tokenizer before starting training.","Keep the converter tool version in sync with the model release."],"tags":["recognition","tokenizer","config","version-mismatch"],"backgroundTag":null,"analyzedSha":"2661c7c0ef5c613e8f93c6e93b2e052399f0f854","analyzedAt":"2026-08-14T20:17:30.180Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}