PaddlePaddle/PaddleOCR · error · ValueError

Found a {token.__class__} in the saved `added_tokens_decoder

Error message

Found a {token.__class__} in the saved `added_tokens_decoder`, should be a dictionary or an AddedToken instance

What it means

When loading a tokenizer, this code reads tokenizer_config.json and rehydrates the added_tokens_decoder mapping: each entry must be a dict (converted to AddedToken via **kwargs) or an AddedToken instance. Any other JSON type (string, list, number) raises ValueError naming the offending class.

Source

Thrown at ppocr/data/imaug/label_ops.py:1990

        )
        self.tokenizer = TokenizerFast.from_file(fast_tokenizer_file)
        added_tokens_decoder = {}
        added_tokens_map = {}

        if tokenizer_config_file is not None:
            with open(
                tokenizer_config_file, encoding="utf-8"
            ) as tokenizer_config_handle:
                init_kwargs = json.load(tokenizer_config_handle)
                if "added_tokens_decoder" in init_kwargs:
                    for idx, token in init_kwargs["added_tokens_decoder"].items():
                        if isinstance(token, dict):
                            token = AddedToken(**token)
                        if isinstance(token, AddedToken):
                            added_tokens_decoder[int(idx)] = token
                            added_tokens_map[str(token)] = token
                        else:
                            raise ValueError(
                                f"Found a {token.__class__} in the saved `added_tokens_decoder`, should be a dictionary or an AddedToken instance"
                            )
                init_kwargs["added_tokens_decoder"] = added_tokens_decoder
                added_tokens_decoder = init_kwargs.pop("added_tokens_decoder", {})
                tokens_to_add = [
                    token
                    for index, token in sorted(
                        added_tokens_decoder.items(), key=lambda x: x[0]
                    )
                    if token not in added_tokens_decoder
                ]
                added_tokens_encoder = self.added_tokens_encoder(added_tokens_decoder)
                encoder = list(added_tokens_encoder.keys()) + [
                    str(token) for token in tokens_to_add
                ]
                tokens_to_add += [
                    token
                    for token in self.all_special_tokens_extended

View on GitHub (pinned to 2661c7c0ef)

Solutions

1. Inspect tokenizer_config.json: python -c 'import json; d=json.load(open("tokenizer_config.json")); print({k: type(v).__name__ for k,v in d.get("added_tokens_decoder",{}).items()})' and fix any non-dict entry to the AddedToken kwargs form {"content": ..., "lstrip": false, ...}
2. Regenerate the config by loading the tokenizer with a matching transformers version and calling tokenizer.save_pretrained(...)
3. Delete a corrupted added_tokens_decoder block only if no custom added tokens are needed, then re-save
4. Restore the original tokenizer files from the model repository you downloaded

Example fix

// before (tokenizer_config.json)
"added_tokens_decoder": { "0": "<pad>" }
// after
"added_tokens_decoder": { "0": { "content": "<pad>", "lstrip": false, "rstrip": false, "single_word": false, "special": true } }

import json
cfg = json.load(open('tokenizer_config.json', encoding='utf-8'))
for idx, tok in cfg.get('added_tokens_decoder', {}).items():
    if not isinstance(tok, dict):
        raise SystemExit(f'added_tokens_decoder[{idx}] must be a dict, got {type(tok).__name__}: {tok!r}')

def is_valid_added_tokens_decoder(cfg: dict) -> bool:
    return all(isinstance(v, dict) and 'content' in v
               for v in cfg.get('added_tokens_decoder', {}).values())

try:
    tok = TokenClass.from_pretrained(path)
except ValueError as e:
    if 'added_tokens_decoder' in str(e):
        raise RuntimeError(f'Corrupted tokenizer_config.json at {path}; regenerate with save_pretrained') from e
    raise

When it happens

Trigger: Loading a tokenizer whose tokenizer_config.json contains an added_tokens_decoder entry like {"0": "<pad>"} instead of {"0": {"content": "<pad>", ...}}, typically a hand-edited or third-party-converted config.

Common situations: Manually editing tokenizer_config.json to add special tokens; using a tokenizer converted by an older/newer tool that writes a flat string map; mixing tokenizer files from different transformers versions.

Related errors

• Token is required. Set PADDLEOCR_ACCESS_TOKEN or pass token
• Destination parent must be a directory: ${parent}
• Expected a JSON response body.
• {path}: missing 'items' array
• Could not parse xcresulttool JSON: {e}.

AI-assisted analysis of PaddlePaddle/PaddleOCR@2661c7c0ef (2026-08-14). Data as JSON: /api/errors/730f9325bada61c6. Report an issue: GitHub.