{"record":{"id":"335a78401c446d39","repo":"hiyouga/LlamaFactory","slug":"stop-words-are-required-to-replace-the-eos-token","errorCode":null,"errorMessage":"Stop words are required to replace the EOS token.","messagePattern":"Stop words are required to replace the EOS token\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/data/template.py","lineNumber":194,"sourceCode":"            return\n\n        is_added = tokenizer.eos_token_id is None\n        num_added_tokens = tokenizer.add_special_tokens({\"eos_token\": eos_token})\n\n        if is_added:\n            logger.info_rank0(f\"Add eos token: {tokenizer.eos_token}.\")\n        else:\n            logger.info_rank0(f\"Replace eos token: {tokenizer.eos_token}.\")\n\n        if num_added_tokens > 0:\n            logger.warning_rank0(\"New tokens have been added, make sure `resize_vocab` is True.\")\n\n    def fix_special_tokens(self, tokenizer: \"PreTrainedTokenizer\") -> None:\n        r\"\"\"Add eos token and pad token to the tokenizer.\"\"\"\n        stop_words = self.stop_words\n        if self.replace_eos:\n            if not stop_words:\n                raise ValueError(\"Stop words are required to replace the EOS token.\")\n\n            self._add_or_replace_eos_token(tokenizer, eos_token=stop_words[0])\n            stop_words = stop_words[1:]\n\n        if tokenizer.eos_token_id is None:\n            self._add_or_replace_eos_token(tokenizer, eos_token=\"<|endoftext|>\")\n\n        if tokenizer.pad_token_id is None:\n            tokenizer.pad_token = tokenizer.eos_token\n            logger.info_rank0(f\"Add pad token: {tokenizer.pad_token}\")\n\n        if stop_words:\n            try:\n                num_added_tokens = tokenizer.add_special_tokens(\n                    dict(additional_special_tokens=stop_words), replace_additional_special_tokens=False\n                )\n            except TypeError:\n                num_added_tokens = tokenizer.add_special_tokens(dict(additional_special_tokens=stop_words))","sourceCodeStart":176,"sourceCodeEnd":212,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/data/template.py#L176-L212","documentation":"Some templates set replace_eos=True to make the tokenizer's EOS a template-specific stop word (the first entry of stop_words). If stop_words is empty or None for such a template, there is nothing to swap in and LlamaFactory raises this ValueError in fix_special_tokens before tokenization starts.","triggerScenarios":"Using a built-in template that declares replace_eos=True but has no stop_words (only possible if the template registry entry was misconstructed, e.g. by a plugin or a modified template); creating a custom template with replace_eos=True while omitting the stop_words argument in register_template; unpickling or copying a Template object whose stop_words were cleared.","commonSituations":"Writing a custom template and setting replace_eos=True without understanding it consumes stop_words[0] as the new EOS; library version changes that renamed the parameter; subclassing Template and overriding stop_words with None.","solutions":["If you authored the template, provide stop_words=[\"<your_eos>\"] so the first entry can replace the EOS token.","If you do not need EOS replacement, remove replace_eos=True from your register_template call.","If using a built-in template, report the misconfiguration upstream — built-in templates with replace_eos always ship stop_words."],"exampleFix":"# before\nregister_template(name=\"custom\", ..., replace_eos=True)\n\n# after\nregister_template(name=\"custom\", ..., stop_words=[\"<|im_end|>\"], replace_eos=True)","handlingStrategy":"validation","validationCode":"def template_eos_config_ok(template) -> bool:\n    return not template.replace_eos or bool(template.stop_words)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["When setting replace_eos=True, always pass stop_words and put the intended EOS first.","Smoke-test get_template_and_fix_tokenizer(tokenizer, DataArguments(template=...)) right after registering a custom template."],"tags":["template","tokenizer","custom-template"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}