opendatalab/MinerU · error · ValueError

Found a {token.__class__} in the saved `added_tokens_decoder

Error message

Found a {token.__class__} in the saved `added_tokens_decoder`, should be a dictionary or an AddedToken instance

What it means

Thrown while loading the tokenizer config for PP-FormulaNet-Plus: each entry of the saved added_tokens_decoder must deserialize into either a dict (which is then converted via AddedToken(**token)) or an AddedToken instance. Any other Python type (str, int, list, None) means the persisted tokenizer_config.json is malformed or was written by an incompatible tokenizers/transformers version.

Source

Thrown at mineru/model/mfr/pp_formulanet_plus_m/processors.py:361

        self.tokenizer = TokenizerFast.from_buffer(fast_tokenizer_buffer)
        tokenizer_config = (
            character_list["tokenizer_config_file"]
            if "tokenizer_config_file" in character_list
            else None
        )
        added_tokens_decoder = {}
        added_tokens_map = {}
        if tokenizer_config is not None:
            init_kwargs = tokenizer_config
            if "added_tokens_decoder" in init_kwargs:
                for idx, token in init_kwargs["added_tokens_decoder"].items():
                    if isinstance(token, dict):
                        token = AddedToken(**token)
                    if isinstance(token, AddedToken):
                        added_tokens_decoder[int(idx)] = token
                        added_tokens_map[str(token)] = token
                    else:
                        raise ValueError(
                            f"Found a {token.__class__} in the saved `added_tokens_decoder`, should be a dictionary or an AddedToken instance"
                        )
            init_kwargs["added_tokens_decoder"] = added_tokens_decoder
            added_tokens_decoder = init_kwargs.pop("added_tokens_decoder", {})
            tokens_to_add = [
                token
                for index, token in sorted(
                    added_tokens_decoder.items(), key=lambda x: x[0]
                )
                if token not in added_tokens_decoder
            ]
            added_tokens_encoder = self.added_tokens_encoder(added_tokens_decoder)
            encoder = list(added_tokens_encoder.keys()) + [
                str(token) for token in tokens_to_add
            ]
            tokens_to_add += [
                token
                for token in self.all_special_tokens_extended

View on GitHub (pinned to 4fe4bde114)

Solutions

1. Inspect tokenizer_config.json's added_tokens_decoder and rewrite each entry as {"content": ..., "lstrip": ..., "rstrip": ..., "single_word": ...}.
2. Re-download the model directory in case the config file is truncated or corrupted.
3. Re-save the tokenizer with the same transformers version used at inference: tokenizer.save_pretrained(dir).
4. Pin transformers/tokenizers to the version the model card for PP-FormulaNet-Plus recommends.

Example fix

// tokenizer_config.json (before)
"added_tokens_decoder": { "0": "<pad>" }

// after
"added_tokens_decoder": {
  "0": {"content": "<pad>", "lstrip": false, "rstrip": false, "single_word": false, "special": true}
}

import json
def validate_added_tokens(path):
    cfg = json.load(open(path))
    for idx, tok in (cfg.get('added_tokens_decoder') or {}).items():
        assert isinstance(tok, dict), f'entry {idx} is {type(tok).__name__}, expected dict'
        assert 'content' in tok, f'entry {idx} missing content'

def is_valid_token_entry(tok) -> bool:
    from tokenizers import AddedToken
    return isinstance(tok, (dict, AddedToken))

try:
    model = AutoTokenizer.from_pretrained(model_dir)
except ValueError as e:
    if 'added_tokens_decoder' in str(e):
        raise RuntimeError(f'tokenizer_config.json in {model_dir} is malformed; re-download or re-save the model') from e
    raise

When it happens

Trigger: Loading a tokenizer whose tokenizer_config.json has added_tokens_decoder entries like "0": "<pad>" (a bare string) instead of "0": {"content": "<pad>", ...}. Also triggered by hand-edited config files or configs generated by a newer transformers version with a changed serialization format.

Common situations: Mixing transformers/hf_tokenizers versions between save and load, manually patching tokenizer configs to add special tokens, or truncated/corrupted JSON after a partial download of the model directory.

Related errors

• config._name_or_path is required by UnimernetModel.
• backend={backend} requires server_url
• effort must be "medium" or "high"
• Unsupported lmdeploy device type: {device_type}
• Unsupported lmdeploy device type: {device_type}

AI-assisted analysis of opendatalab/MinerU@4fe4bde114 (2026-08-14). Data as JSON: /api/errors/e72aec0c64b15424. Report an issue: GitHub.