PaddlePaddle/PaddleOCR · error · ValueError

Found a {token.__class__} in the saved `added_tokens_decoder

Error message

Found a {token.__class__} in the saved `added_tokens_decoder`, should be a dictionary or an AddedToken instance

What it means

ValueError from the vendored transformers-style tokenizer loading in rec_postprocess.py: while replaying tokenizer_config.json's added_tokens_decoder, an entry is neither a dict nor an AddedToken instance. The saved file is malformed or was produced/consumed by a library with an incompatible added_tokens_decoder schema.

Source

Thrown at ppocr/postprocess/rec_postprocess.py:1344

            rec_char_dict_path, "tokenizer_config.json"
        )
        self.tokenizer = TokenizerFast.from_file(fast_tokenizer_file)
        added_tokens_decoder = {}
        added_tokens_map = {}
        if tokenizer_config_file is not None:
            with open(
                tokenizer_config_file, encoding="utf-8"
            ) as tokenizer_config_handle:
                init_kwargs = json.load(tokenizer_config_handle)
                if "added_tokens_decoder" in init_kwargs:
                    for idx, token in init_kwargs["added_tokens_decoder"].items():
                        if isinstance(token, dict):
                            token = AddedToken(**token)
                        if isinstance(token, AddedToken):
                            added_tokens_decoder[int(idx)] = token
                            added_tokens_map[str(token)] = token
                        else:
                            raise ValueError(
                                f"Found a {token.__class__} in the saved `added_tokens_decoder`, should be a dictionary or an AddedToken instance"
                            )
                init_kwargs["added_tokens_decoder"] = added_tokens_decoder
                added_tokens_decoder = init_kwargs.pop("added_tokens_decoder", {})
                tokens_to_add = [
                    token
                    for index, token in sorted(
                        added_tokens_decoder.items(), key=lambda x: x[0]
                    )
                    if token not in added_tokens_decoder
                ]
                added_tokens_encoder = self.added_tokens_encoder(added_tokens_decoder)
                encoder = list(added_tokens_encoder.keys()) + [
                    str(token) for token in tokens_to_add
                ]
                tokens_to_add += [
                    token
                    for token in self.all_special_tokens_extended

View on GitHub (pinned to 2661c7c0ef)

Solutions

1. Regenerate or re-download the matching tokenizer_config.json from the official model release instead of hand-editing it.
2. Inspect added_tokens_decoder in the file: each value must be an object like {"content": "<unk>", "lstrip": false, ...} or an AddedToken-equivalent dict.
3. Pin the model repo/converter version that produced the tokenizer so the schema matches the vendored loader.
4. If a value is a plain string, wrap it: {"content": "<your_token>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": false}.

Example fix

// tokenizer_config.json before
"added_tokens_decoder": {"0": "<pad>"}

// after
"added_tokens_decoder": {"0": {"content": "<pad>", "lstrip": false, "rstrip": false, "single_word": false, "normalized": false}}

import json

def valid_tokenizer_config(path) -> bool:
    with open(path, encoding='utf-8') as f:
        cfg = json.load(f)
    atd = cfg.get('added_tokens_decoder', {})
    return all(isinstance(v, dict) for v in atd.values())

def is_valid_added_tokens_decoder(obj) -> bool:
    return (
        isinstance(obj, dict)
        and all(str(k).isdigit() for k in obj)
        and all(isinstance(v, dict) for v in obj.values())
    )

When it happens

Trigger: Initializing the recognition postprocess (e.g. for LaTeX/VLrec-type models like UniMERNet or GenRec tokenizer-based decoding) with a tokenizer_config.json whose added_tokens_decoder maps indices to plain strings, nulls, or serialized non-dict objects.

Common situations: Mixing tokenizer file versions: config generated by a newer/older transformers release than the vendored reader expects; hand-merged tokenizer directories; partial downloads truncating the JSON into odd structures; converting checkpoints with third-party scripts that rewrite the file.

Related errors

• RecResizeImg.image_shape is required in rec inference.yml
• Unexpected recognition channels: ${String(channels)}
• {value} is not a valid {cls.__name__}, please select one of
• Unsupported interpolation type !!!
• {} is not supported in MultiLoss yet

AI-assisted analysis of PaddlePaddle/PaddleOCR@2661c7c0ef (2026-08-14). Data as JSON: /api/errors/424906b42ada6901. Report an issue: GitHub.