{"record":{"id":"8211a01e9eb9cfc3","repo":"hiyouga/LlamaFactory","slug":"yaml-config-must-be-a-dictionary-mapping-tokens-to","errorCode":null,"errorMessage":"YAML config must be a dictionary mapping tokens to descriptions. Got: {type(token_descriptions)}","messagePattern":"YAML config must be a dictionary mapping tokens to descriptions\\. Got: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/model_args.py","lineNumber":224,"sourceCode":"    def __post_init__(self):\n        if self.model_name_or_path is None:\n            raise ValueError(\"Please provide `model_name_or_path`.\")\n\n        if self.adapter_name_or_path is not None:  # support merging multiple lora weights\n            self.adapter_name_or_path = [path.strip() for path in self.adapter_name_or_path.split(\",\")]\n\n        if self.add_tokens is not None:  # support multiple tokens\n            self.add_tokens = [token.strip() for token in self.add_tokens.split(\",\")]\n\n        # Process special tokens with priority: new_special_tokens_config > add_special_tokens\n        if self.new_special_tokens_config is not None:\n            # Priority 1: Load from YAML config (extracts both tokens and descriptions)\n            try:\n                cfg = OmegaConf.load(self.new_special_tokens_config)\n                token_descriptions = OmegaConf.to_container(cfg)\n\n                if not isinstance(token_descriptions, dict):\n                    raise ValueError(\n                        f\"YAML config must be a dictionary mapping tokens to descriptions. \"\n                        f\"Got: {type(token_descriptions)}\"\n                    )\n\n                # Extract token list from config keys\n                extracted_tokens = list(token_descriptions.keys())\n\n                # Warn if both are set\n                if self.add_special_tokens is not None:\n                    logger.warning_rank0(\n                        \"Both 'new_special_tokens_config' and 'add_special_tokens' are set. \"\n                        f\"Using tokens from config: {extracted_tokens}\"\n                    )\n\n                # Override add_special_tokens with extracted tokens (as list)\n                self.add_special_tokens = extracted_tokens\n\n                # Store descriptions internally for later use (internal attribute)","sourceCodeStart":206,"sourceCodeEnd":242,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/model_args.py#L206-L242","documentation":"Raised in ModelArguments.__post_init__ (model_args.py:224) when new_special_tokens_config points to a YAML file whose top-level structure is not a mapping of token -> description. The config is loaded with OmegaConf and converted with to_container; if the result is a list, scalar, or null (e.g. the file contains only a '- token' list or a bare string), the isinstance(dict) check fails and this ValueError is raised during argument parsing.","triggerScenarios":"new_special_tokens_config: specials.yaml where specials.yaml contains a YAML list of tokens or plain text instead of key: value pairs; an empty YAML file (to_container gives None); a JSON file whose top level is an array.","commonSituations":"Users writing a token list (the intuitive format) instead of a mapping; reusing the same file for add_special_tokens (comma string) and the config variant; hand-editing that accidentally deletes the mapping structure.","solutions":["Rewrite the YAML as a mapping: each top-level key is the token string, its value the description","Example: '<|im_start|>': 'start of turn' as top-level entries, no leading dashes","If you only need tokens without descriptions, use the simpler add_special_tokens: '<tok1>,<tok2>' comma-separated string instead"],"exampleFix":"# before (specials.yaml)\n- <|im_start|>\n- <|im_end|>\n\n# after (specials.yaml)\n'<|im_start|>': 'start of turn'\n'<|im_end|>': 'end of turn'","handlingStrategy":"type-guard","validationCode":"from omegaconf import OmegaConf\ncfg_doc = OmegaConf.to_container(OmegaConf.load(path))\nassert isinstance(cfg_doc, dict), 'special tokens YAML must be a token->description mapping'","typeGuard":"def is_token_mapping(v: object) -> bool:\n    return isinstance(v, dict) and all(isinstance(k, str) for k in v)","tryCatchPattern":"try:\n    args = ModelArguments(**cfg)\nexcept ValueError as e:\n    if 'YAML config must be a dictionary' in str(e):\n        rewrite_list_to_mapping(path)  # '- tok' -> \"'tok': desc\"\n        args = ModelArguments(**cfg)\n    else:\n        raise","preventionTips":["Token-only needs: use add_special_tokens comma string instead of the config file","Keep one canonical example mapping file in the repo and diff against it"],"tags":["hparams","tokenizer","special-tokens","yaml","config-validation"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}