{"record":{"id":"aff54d42db4b1638","repo":"invoke-ai/InvokeAI","slug":"token-ids-must-not-end-with-eos-token-id","errorCode":null,"errorMessage":"token_ids must not end with eos_token_id","messagePattern":"token_ids must not end with eos_token_id","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"invokeai/backend/textual_inversion.py","lineNumber":111,"sourceCode":"        ```\n        <ti_dog>: 49408\n        <ti_dog-!pad-1>: 49409\n        <ti_dog-!pad-2>: 49410\n        <ti_dog-!pad-3>: 49411\n        ```\n        `self.pad_tokens` would be set to `{49408: [49408, 49409, 49410, 49411]}`.\n        This function is responsible for expanding `49408` in the token_ids list to `[49408, 49409, 49410, 49411]`.\n        \"\"\"\n        # Short circuit if there are no pad tokens to save a little time.\n        if len(self.pad_tokens) == 0:\n            return token_ids\n\n        # This function assumes that compel has not included the BOS and EOS tokens in the token_ids list. We verify\n        # this assumption here.\n        if token_ids[0] == self.tokenizer.bos_token_id:\n            raise ValueError(\"token_ids must not start with bos_token_id\")\n        if token_ids[-1] == self.tokenizer.eos_token_id:\n            raise ValueError(\"token_ids must not end with eos_token_id\")\n\n        # Expand any TI tokens to their corresponding pad tokens.\n        new_token_ids: list[int] = []\n        for token_id in token_ids:\n            new_token_ids.append(token_id)\n            if token_id in self.pad_tokens:\n                new_token_ids.extend(self.pad_tokens[token_id])\n\n        # Do not exceed the max model input size. The -2 here is compensating for\n        # compel.embeddings_provider.get_token_ids(), which first removes and then adds back the start and end tokens.\n        max_length = self.tokenizer.model_max_length - 2\n        if len(new_token_ids) > max_length:\n            # HACK: If TI token expansion causes us to exceed the max text encoder input length, we silently discard\n            # tokens. Token expansion should happen in a way that is compatible with compel's default handling of long\n            # prompts.\n            new_token_ids = new_token_ids[0:max_length]\n\n        return new_token_ids","sourceCodeStart":93,"sourceCodeEnd":129,"githubUrl":"https://github.com/invoke-ai/InvokeAI/blob/0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06/invokeai/backend/textual_inversion.py#L93-L129","documentation":"The mirror-image guard of the BOS check: the same function assumes compel has not appended EOS, and raises if token_ids[-1] equals tokenizer.eos_token_id. It ensures pad-token expansion produces ids that the model's encoder can wrap in specials exactly once.","triggerScenarios":"Calling expand_textual_inversion_token_ids_if_necessary() with ids whose last element is the tokenizer's eos_token_id — e.g. ids from tokenizer(...) with specials included, or ids copied from compel output after a version change re-enabled specials.","commonSituations":"Same as the BOS case: hand-rolled tokenization with add_special_tokens=True, pipeline refactors passing full encoded sequences into the TI manager, or custom prompt-builder that appends EOS itself.","solutions":["Strip the trailing EOS before calling: token_ids[:-1] if token_ids[-1] == tokenizer.eos_token_id else token_ids","Encode with add_special_tokens=False and add specials only at the final model input stage","Align with your compel version's behavior for included special tokens"],"exampleFix":"// before\ntoken_ids = tokenizer(prompt)[\"input_ids\"]  # ends with EOS\nids = ti_manager.expand_textual_inversion_token_ids_if_necessary(token_ids)\n// after\ntoken_ids = tokenizer(prompt, add_special_tokens=False)[\"input_ids\"]\nids = ti_manager.expand_textual_inversion_token_ids_if_necessary(token_ids)","handlingStrategy":"validation","validationCode":"ids = tokenizer(text, add_special_tokens=False)[\"input_ids\"]\nassert ids[-1] != tokenizer.eos_token_id, \"strip EOS before TI expansion\"\nids = ti_manager.expand_textual_inversion_token_ids_if_necessary(ids)","typeGuard":"def is_eos_free(ids: list[int], tokenizer) -> bool:\n    return len(ids) > 0 and ids[-1] != tokenizer.eos_token_id","tryCatchPattern":"try:\n    ids = ti_manager.expand_textual_inversion_token_ids_if_necessary(token_ids)\nexcept ValueError as e:\n    if \"must not end with eos_token_id\" in str(e):\n        token_ids = token_ids[:-1]\n        ids = ti_manager.expand_textual_inversion_token_ids_if_necessary(token_ids)\n    else:\n        raise","preventionTips":["Strip trailing EOS before calling the TI expansion","Use add_special_tokens=False when tokenizing fragments","Add assertions that specials are absent before pad-token expansion","Keep tokenizer/compel versions consistent with the pipeline's assumptions"],"tags":["tokenization","textual-inversion","validation","contract-violation"],"backgroundTag":"unexpected-special-token","analyzedSha":"0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06","analyzedAt":"2026-08-29T04:46:49.967Z","schemaVersion":2},"datasetVersion":"2026-08-29T07:17:48.351Z"}