invoke-ai/InvokeAI · error · ValueError

token_ids must not end with eos_token_id

Error message

token_ids must not end with eos_token_id

What it means

The mirror-image guard of the BOS check: the same function assumes compel has not appended EOS, and raises if token_ids[-1] equals tokenizer.eos_token_id. It ensures pad-token expansion produces ids that the model's encoder can wrap in specials exactly once.

Source

Thrown at invokeai/backend/textual_inversion.py:111

        ```
        <ti_dog>: 49408
        <ti_dog-!pad-1>: 49409
        <ti_dog-!pad-2>: 49410
        <ti_dog-!pad-3>: 49411
        ```
        `self.pad_tokens` would be set to `{49408: [49408, 49409, 49410, 49411]}`.
        This function is responsible for expanding `49408` in the token_ids list to `[49408, 49409, 49410, 49411]`.
        """
        # Short circuit if there are no pad tokens to save a little time.
        if len(self.pad_tokens) == 0:
            return token_ids

        # This function assumes that compel has not included the BOS and EOS tokens in the token_ids list. We verify
        # this assumption here.
        if token_ids[0] == self.tokenizer.bos_token_id:
            raise ValueError("token_ids must not start with bos_token_id")
        if token_ids[-1] == self.tokenizer.eos_token_id:
            raise ValueError("token_ids must not end with eos_token_id")

        # Expand any TI tokens to their corresponding pad tokens.
        new_token_ids: list[int] = []
        for token_id in token_ids:
            new_token_ids.append(token_id)
            if token_id in self.pad_tokens:
                new_token_ids.extend(self.pad_tokens[token_id])

        # Do not exceed the max model input size. The -2 here is compensating for
        # compel.embeddings_provider.get_token_ids(), which first removes and then adds back the start and end tokens.
        max_length = self.tokenizer.model_max_length - 2
        if len(new_token_ids) > max_length:
            # HACK: If TI token expansion causes us to exceed the max text encoder input length, we silently discard
            # tokens. Token expansion should happen in a way that is compatible with compel's default handling of long
            # prompts.
            new_token_ids = new_token_ids[0:max_length]

        return new_token_ids

View on GitHub (pinned to 0b6a024f2f)

Solutions

1. Strip the trailing EOS before calling: token_ids[:-1] if token_ids[-1] == tokenizer.eos_token_id else token_ids
2. Encode with add_special_tokens=False and add specials only at the final model input stage
3. Align with your compel version's behavior for included special tokens

Example fix

// before
token_ids = tokenizer(prompt)["input_ids"]  # ends with EOS
ids = ti_manager.expand_textual_inversion_token_ids_if_necessary(token_ids)
// after
token_ids = tokenizer(prompt, add_special_tokens=False)["input_ids"]
ids = ti_manager.expand_textual_inversion_token_ids_if_necessary(token_ids)

ids = tokenizer(text, add_special_tokens=False)["input_ids"]
assert ids[-1] != tokenizer.eos_token_id, "strip EOS before TI expansion"
ids = ti_manager.expand_textual_inversion_token_ids_if_necessary(ids)

def is_eos_free(ids: list[int], tokenizer) -> bool:
    return len(ids) > 0 and ids[-1] != tokenizer.eos_token_id

try:
    ids = ti_manager.expand_textual_inversion_token_ids_if_necessary(token_ids)
except ValueError as e:
    if "must not end with eos_token_id" in str(e):
        token_ids = token_ids[:-1]
        ids = ti_manager.expand_textual_inversion_token_ids_if_necessary(token_ids)
    else:
        raise

When it happens

Trigger: Calling expand_textual_inversion_token_ids_if_necessary() with ids whose last element is the tokenizer's eos_token_id — e.g. ids from tokenizer(...) with specials included, or ids copied from compel output after a version change re-enabled specials.

Common situations: Same as the BOS case: hand-rolled tokenization with add_special_tokens=True, pipeline refactors passing full encoded sequences into the TI manager, or custom prompt-builder that appends EOS itself.

Related errors

• token_ids must not start with bos_token_id
• No external provider config fields provided
• str(e)
• str(e) (ValueError from user service update, e.g. LastAdmini
• Current password is required to set a new password

AI-assisted analysis of invoke-ai/InvokeAI@0b6a024f2f (2026-08-29). Data as JSON: /api/errors/aff54d42db4b1638. Report an issue: GitHub.