{"record":{"id":"8481a4935291bbf5","repo":"invoke-ai/InvokeAI","slug":"unable-to-find-token-id-for-token-trigger","errorCode":null,"errorMessage":"Unable to find token id for token '{trigger}'","messagePattern":"Unable to find token id for token '(.+?)'","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"invokeai/backend/model_patcher.py","lineNumber":120,"sourceCode":"            # resize_token_embeddings(...) constructs a new torch.nn.Embedding internally. Initializing the weights of\n            # this embedding is slow and unnecessary, so we wrap this step in skip_torch_weight_init() to save some\n            # time.\n            with skip_torch_weight_init():\n                text_encoder.resize_token_embeddings(init_tokens_count + new_tokens_added, pad_to_multiple_of)\n            model_embeddings = text_encoder.get_input_embeddings()\n\n            for ti_name, ti in ti_list:\n                assert isinstance(ti, TextualInversionModelRaw)\n                ti_embedding = _get_ti_embedding(text_encoder.get_input_embeddings(), ti)\n\n                ti_tokens = []\n                for i in range(ti_embedding.shape[0]):\n                    embedding = ti_embedding[i]\n                    trigger = _get_trigger(ti_name, i)\n\n                    token_id = ti_tokenizer.convert_tokens_to_ids(trigger)\n                    if token_id == ti_tokenizer.unk_token_id:\n                        raise RuntimeError(f\"Unable to find token id for token '{trigger}'\")\n\n                    if model_embeddings.weight.data[token_id].shape != embedding.shape:\n                        raise ValueError(\n                            f\"Cannot load embedding for {trigger}. It was trained on a model with token dimension\"\n                            f\" {embedding.shape[0]}, but the current model has token dimension\"\n                            f\" {model_embeddings.weight.data[token_id].shape[0]}.\"\n                        )\n\n                    model_embeddings.weight.data[token_id] = embedding.to(\n                        device=TorchDevice.choose_torch_device(), dtype=text_encoder.dtype\n                    )\n                    ti_tokens.append(token_id)\n\n                if len(ti_tokens) > 1:\n                    ti_manager.pad_tokens[ti_tokens[0]] = ti_tokens[1:]\n\n            yield ti_tokenizer, ti_manager\n","sourceCodeStart":102,"sourceCodeEnd":138,"githubUrl":"https://github.com/invoke-ai/InvokeAI/blob/0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06/invokeai/backend/model_patcher.py#L102-L138","documentation":"apply_ti() injects textual-inversion embedding vectors into the CLIP token embedding matrix. It maps each trigger token via the tokenizer; if the tokenizer returns unk_token_id the trigger does not exist in the model's vocabulary, so patching cannot proceed and RuntimeError is raised.","triggerScenarios":"Loading a textual inversion embedding whose trigger token (from _get_trigger, e.g. '<my-embedding>' or per-vector triggers) is not resolvable to a token id by ti_tokenizer — typically because the embedding's trigger strings were never added as special tokens to the tokenizer before this call.","commonSituations":"Embeddings trained for a different tokenizer/CLIP variant than the loaded model; mismatched embedding naming so trigger lookup falls back to the raw name; corrupted embedding metadata lacking trained_tokens.","solutions":["Verify the embedding's trigger string matches what you typed in the prompt and that the embedding is loaded (check InvokeAI's embedding list).","Use an embedding trained for the same base model/tokenizer (e.g. SD1.5 embedding on SD1.5, not SDXL).","Re-import the embedding so its trigger tokens are registered; ensure the embedding file metadata contains valid trigger names.","Update InvokeAI — token registration for embeddings has been fixed in several releases."],"exampleFix":"// before: trigger unknown to tokenizer\ntoken_id = ti_tokenizer.convert_tokens_to_ids(trigger)\n// after: register the trigger as a special token before patching\nnum_added = ti_tokenizer.add_tokens(trigger)\ntext_encoder.resize_token_embeddings(len(ti_tokenizer))","handlingStrategy":"validation","validationCode":"triggers = ti_tokenizer.convert_tokens_to_ids(trigger)\nif triggers == ti_tokenizer.unk_token_id:\n    print(f'Trigger {trigger!r} not in vocabulary; embedding incompatible with this tokenizer')\n    # register it: ti_tokenizer.add_tokens(trigger) + resize embeddings","typeGuard":"def trigger_exists(tokenizer, trigger: str) -> bool:\n    return tokenizer.convert_tokens_to_ids(trigger) != tokenizer.unk_token_id","tryCatchPattern":"try:\n    patcher.apply_ti(...)\nexcept RuntimeError as e:\n    if str(e).startswith('Unable to find token id'):\n        missing = str(e).split(\"'\")[1]\n        print(f'Embedding trigger {missing} not registered for this model/tokenizer')\n    else:\n        raise","preventionTips":["Match embeddings to the base model's tokenizer (SD1.5 vs SD2.x vs SDXL).","Confirm the exact trigger string via the Model Manager's embedding info before prompting.","Keep InvokeAI updated so embedding trigger registration is handled automatically."],"tags":["textual-inversion","tokenizer","embedding"],"backgroundTag":"unknown-token-id","analyzedSha":"0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06","analyzedAt":"2026-08-29T04:46:49.967Z","schemaVersion":2},"datasetVersion":"2026-08-29T07:17:48.351Z"}