{"record":{"id":"0b1bd5cb9662fbb6","repo":"invoke-ai/InvokeAI","slug":"token-ids-must-not-start-with-bos-token-id","errorCode":null,"errorMessage":"token_ids must not start with bos_token_id","messagePattern":"token_ids must not start with bos_token_id","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"invokeai/backend/textual_inversion.py","lineNumber":109,"sourceCode":"        For example, suppose we have a `<ti_dog>` TI with 4 vectors that was added to the tokenizer with the following\n        mapping of tokens to token_ids:\n        ```\n        <ti_dog>: 49408\n        <ti_dog-!pad-1>: 49409\n        <ti_dog-!pad-2>: 49410\n        <ti_dog-!pad-3>: 49411\n        ```\n        `self.pad_tokens` would be set to `{49408: [49408, 49409, 49410, 49411]}`.\n        This function is responsible for expanding `49408` in the token_ids list to `[49408, 49409, 49410, 49411]`.\n        \"\"\"\n        # Short circuit if there are no pad tokens to save a little time.\n        if len(self.pad_tokens) == 0:\n            return token_ids\n\n        # This function assumes that compel has not included the BOS and EOS tokens in the token_ids list. We verify\n        # this assumption here.\n        if token_ids[0] == self.tokenizer.bos_token_id:\n            raise ValueError(\"token_ids must not start with bos_token_id\")\n        if token_ids[-1] == self.tokenizer.eos_token_id:\n            raise ValueError(\"token_ids must not end with eos_token_id\")\n\n        # Expand any TI tokens to their corresponding pad tokens.\n        new_token_ids: list[int] = []\n        for token_id in token_ids:\n            new_token_ids.append(token_id)\n            if token_id in self.pad_tokens:\n                new_token_ids.extend(self.pad_tokens[token_id])\n\n        # Do not exceed the max model input size. The -2 here is compensating for\n        # compel.embeddings_provider.get_token_ids(), which first removes and then adds back the start and end tokens.\n        max_length = self.tokenizer.model_max_length - 2\n        if len(new_token_ids) > max_length:\n            # HACK: If TI token expansion causes us to exceed the max text encoder input length, we silently discard\n            # tokens. Token expansion should happen in a way that is compatible with compel's default handling of long\n            # prompts.\n            new_token_ids = new_token_ids[0:max_length]","sourceCodeStart":91,"sourceCodeEnd":127,"githubUrl":"https://github.com/invoke-ai/InvokeAI/blob/0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06/invokeai/backend/textual_inversion.py#L91-L127","documentation":"expand_textual_inversion_token_ids_if_necessary assumes compel has NOT included special BOS/EOS tokens in token_ids; it asserts that assumption before expanding textual-inversion tokens to pad tokens. Passing ids that begin with the tokenizer's bos_token_id violates the contract and raises this ValueError.","triggerScenarios":"Calling TextualInversionManager.expand_textual_inversion_token_ids_if_necessary(token_ids) with a list whose first element equals tokenizer.bos_token_id — i.e. ids produced by a full tokenizer() call instead of compel's encode (which strips specials).","commonSituations":"Manually tokenizing prompt fragments with add_special_tokens=True and feeding them to the TI expansion; swapping compel versions so specials are included; double-encoding already-processed ids.","solutions":["Pass raw token ids without BOS/EOS (encode with add_special_tokens=False or use compel's encode output)","Strip leading/trailing special tokens before calling the expansion: token_ids = ids[1:-1] if needed","Check your compel version's contract on whether specials are included"],"exampleFix":"// before\ntoken_ids = tokenizer(prompt)[\"input_ids\"]  # includes BOS\nids = ti_manager.expand_textual_inversion_token_ids_if_necessary(token_ids)\n// after\ntoken_ids = tokenizer(prompt, add_special_tokens=False)[\"input_ids\"]\nids = ti_manager.expand_textual_inversion_token_ids_if_necessary(token_ids)","handlingStrategy":"validation","validationCode":"ids = tokenizer(text, add_special_tokens=False)[\"input_ids\"]\nassert ids[0] != tokenizer.bos_token_id, \"strip BOS before TI expansion\"\nids = ti_manager.expand_textual_inversion_token_ids_if_necessary(ids)","typeGuard":"def is_bos_free(ids: list[int], tokenizer) -> bool:\n    return len(ids) > 0 and ids[0] != tokenizer.bos_token_id","tryCatchPattern":"try:\n    ids = ti_manager.expand_textual_inversion_token_ids_if_necessary(token_ids)\nexcept ValueError as e:\n    if \"must not start with bos_token_id\" in str(e):\n        token_ids = token_ids[1:]\n        ids = ti_manager.expand_textual_inversion_token_ids_if_necessary(token_ids)\n    else:\n        raise","preventionTips":["Always encode with add_special_tokens=False before TI expansion","Never feed tokenizer(prompt)[\"input_ids\"] directly into the TI manager","Pin your compel version and read its contract on special tokens","Assert BOS/EOS absence in prompt-preprocessing unit tests"],"tags":["tokenization","textual-inversion","validation","contract-violation"],"backgroundTag":"unexpected-special-token","analyzedSha":"0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06","analyzedAt":"2026-08-29T04:46:49.967Z","schemaVersion":2},"datasetVersion":"2026-08-29T07:17:48.351Z"}