{"record":{"id":"7658adeebe5d998d","repo":"invoke-ai/InvokeAI","slug":"tokenizer-returned-unexpected-types","errorCode":null,"errorMessage":"Tokenizer returned unexpected types.","messagePattern":"Tokenizer returned unexpected types\\.","errorType":"exception","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"invokeai/app/invocations/anima_text_encoder.py","lineNumber":166,"sourceCode":"                raise TypeError(f\"Expected PreTrainedTokenizerBase for tokenizer, got {type(tokenizer).__name__}.\")\n\n            context.util.signal_progress(\"Running Qwen3 0.6B text encoder\")\n\n            # Anima uses base Qwen3 (not instruct) — tokenize directly, no chat template.\n            # A safety cap is applied to prevent GPU OOM on extremely long prompts.\n            text_inputs = tokenizer(\n                prompt,\n                padding=False,\n                truncation=True,\n                max_length=QWEN3_MAX_SEQ_LEN,\n                return_attention_mask=True,\n                return_tensors=\"pt\",\n            )\n\n            text_input_ids = text_inputs.input_ids\n            attention_mask = text_inputs.attention_mask\n            if not isinstance(text_input_ids, torch.Tensor) or not isinstance(attention_mask, torch.Tensor):\n                raise TypeError(\"Tokenizer returned unexpected types.\")\n\n            if text_input_ids.shape[-1] == QWEN3_MAX_SEQ_LEN:\n                logger.warning(\n                    f\"Prompt was truncated to {QWEN3_MAX_SEQ_LEN} tokens. \"\n                    \"Consider shortening the prompt for best results.\"\n                )\n\n            # Ensure at least 1 token (empty prompts produce 0 tokens with padding=False)\n            if text_input_ids.shape[-1] == 0:\n                pad_id = tokenizer.pad_token_id if tokenizer.pad_token_id is not None else tokenizer.eos_token_id\n                text_input_ids = torch.tensor([[pad_id]])\n                attention_mask = torch.tensor([[1]])\n\n            # Get last hidden state from Qwen3 (final layer output)\n            prompt_mask = attention_mask.to(device).bool()\n            outputs = text_encoder(\n                text_input_ids.to(device),\n                attention_mask=prompt_mask,","sourceCodeStart":148,"sourceCodeEnd":184,"githubUrl":"https://github.com/invoke-ai/InvokeAI/blob/0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06/invokeai/app/invocations/anima_text_encoder.py#L148-L184","documentation":"After calling the tokenizer, _encode_prompt verifies that input_ids and attention_mask are torch.Tensors. If the tokenizer returns lists, None, or other container types instead, it raises a TypeError, since downstream code indexes into tensor shapes and moves them to device.","triggerScenarios":"During invoke → _encode_prompt, when the Qwen3 tokenizer call returns a BatchEncoding whose input_ids or attention_mask are not torch tensors — e.g. return_tensors='pt' was not honored, a custom/subclassed tokenizer returned lists, or a stub tokenizer returned None.","commonSituations":"A tokenizer monkey-patched or wrapped by another integration; transformers version where the tokenizer returns nested lists for edge inputs; testing with a mock tokenizer that doesn't emulate tensor conversion.","solutions":["Use the standard transformers tokenizer for the model, not a custom subclass or wrapper.","Explicitly convert with torch.tensor(text_inputs.input_ids) before proceeding if a custom tokenizer must be used.","Ensure the transformers version matches the installed InvokeAI requirements."],"exampleFix":"// before\ntext_inputs = tokenizer(prompt, return_tensors=\"pt\")  # custom tokenizer returns lists\n// after\ntext_inputs = tokenizer(prompt, return_tensors=\"pt\")\nif not isinstance(text_inputs.input_ids, torch.Tensor):\n    text_inputs.input_ids = torch.tensor(text_inputs.input_ids)","handlingStrategy":"type-guard","validationCode":"import torch\nout = tokenizer(prompt, return_tensors=\"pt\")\nassert isinstance(out.input_ids, torch.Tensor) and isinstance(out.attention_mask, torch.Tensor)","typeGuard":"import torch\ndef tokenizer_output_ok(out) -> bool:\n    return isinstance(getattr(out, \"input_ids\", None), torch.Tensor) and isinstance(getattr(out, \"attention_mask\", None), torch.Tensor)","tryCatchPattern":"try:\n    result = invocation.invoke(context)\nexcept TypeError as e:\n    if \"Tokenizer returned unexpected types\" in str(e):\n        swap_to_stock_transformers_tokenizer()\n    else:\n        raise","preventionTips":["Use the stock transformers tokenizer; avoid subclasses/wrappers that change return types.","Pin the transformers version InvokeAI expects.","If mocking tokenizers in tests, return BatchEncoding with tensor fields."],"tags":["tokenizer","type-mismatch","pytorch","invokeai"],"backgroundTag":"unexpected-tokenizer-output","analyzedSha":"0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06","analyzedAt":"2026-08-29T04:46:49.967Z","schemaVersion":2},"datasetVersion":"2026-08-29T07:17:48.351Z"}