invoke-ai/InvokeAI · error · ValueError

token_ids must not start with bos_token_id

Error message

token_ids must not start with bos_token_id

What it means

expand_textual_inversion_token_ids_if_necessary assumes compel has NOT included special BOS/EOS tokens in token_ids; it asserts that assumption before expanding textual-inversion tokens to pad tokens. Passing ids that begin with the tokenizer's bos_token_id violates the contract and raises this ValueError.

Source

Thrown at invokeai/backend/textual_inversion.py:109

        For example, suppose we have a `<ti_dog>` TI with 4 vectors that was added to the tokenizer with the following
        mapping of tokens to token_ids:
        ```
        <ti_dog>: 49408
        <ti_dog-!pad-1>: 49409
        <ti_dog-!pad-2>: 49410
        <ti_dog-!pad-3>: 49411
        ```
        `self.pad_tokens` would be set to `{49408: [49408, 49409, 49410, 49411]}`.
        This function is responsible for expanding `49408` in the token_ids list to `[49408, 49409, 49410, 49411]`.
        """
        # Short circuit if there are no pad tokens to save a little time.
        if len(self.pad_tokens) == 0:
            return token_ids

        # This function assumes that compel has not included the BOS and EOS tokens in the token_ids list. We verify
        # this assumption here.
        if token_ids[0] == self.tokenizer.bos_token_id:
            raise ValueError("token_ids must not start with bos_token_id")
        if token_ids[-1] == self.tokenizer.eos_token_id:
            raise ValueError("token_ids must not end with eos_token_id")

        # Expand any TI tokens to their corresponding pad tokens.
        new_token_ids: list[int] = []
        for token_id in token_ids:
            new_token_ids.append(token_id)
            if token_id in self.pad_tokens:
                new_token_ids.extend(self.pad_tokens[token_id])

        # Do not exceed the max model input size. The -2 here is compensating for
        # compel.embeddings_provider.get_token_ids(), which first removes and then adds back the start and end tokens.
        max_length = self.tokenizer.model_max_length - 2
        if len(new_token_ids) > max_length:
            # HACK: If TI token expansion causes us to exceed the max text encoder input length, we silently discard
            # tokens. Token expansion should happen in a way that is compatible with compel's default handling of long
            # prompts.
            new_token_ids = new_token_ids[0:max_length]

View on GitHub (pinned to 0b6a024f2f)

Solutions

1. Pass raw token ids without BOS/EOS (encode with add_special_tokens=False or use compel's encode output)
2. Strip leading/trailing special tokens before calling the expansion: token_ids = ids[1:-1] if needed
3. Check your compel version's contract on whether specials are included

Example fix

// before
token_ids = tokenizer(prompt)["input_ids"]  # includes BOS
ids = ti_manager.expand_textual_inversion_token_ids_if_necessary(token_ids)
// after
token_ids = tokenizer(prompt, add_special_tokens=False)["input_ids"]
ids = ti_manager.expand_textual_inversion_token_ids_if_necessary(token_ids)

ids = tokenizer(text, add_special_tokens=False)["input_ids"]
assert ids[0] != tokenizer.bos_token_id, "strip BOS before TI expansion"
ids = ti_manager.expand_textual_inversion_token_ids_if_necessary(ids)

def is_bos_free(ids: list[int], tokenizer) -> bool:
    return len(ids) > 0 and ids[0] != tokenizer.bos_token_id

try:
    ids = ti_manager.expand_textual_inversion_token_ids_if_necessary(token_ids)
except ValueError as e:
    if "must not start with bos_token_id" in str(e):
        token_ids = token_ids[1:]
        ids = ti_manager.expand_textual_inversion_token_ids_if_necessary(token_ids)
    else:
        raise

When it happens

Trigger: Calling TextualInversionManager.expand_textual_inversion_token_ids_if_necessary(token_ids) with a list whose first element equals tokenizer.bos_token_id — i.e. ids produced by a full tokenizer() call instead of compel's encode (which strips specials).

Common situations: Manually tokenizing prompt fragments with add_special_tokens=True and feeding them to the TI expansion; swapping compel versions so specials are included; double-encoding already-processed ids.

Related errors

• token_ids must not end with eos_token_id
• No external provider config fields provided
• str(e)
• str(e) (ValueError from user service update, e.g. LastAdmini
• Current password is required to set a new password

AI-assisted analysis of invoke-ai/InvokeAI@0b6a024f2f (2026-08-29). Data as JSON: /api/errors/0b1bd5cb9662fbb6. Report an issue: GitHub.