What causes "Tokenizer returned unexpected types." in invoke-ai/InvokeAI?

Use the standard transformers tokenizer for the model, not a custom subclass or wrapper.

invoke-ai/InvokeAI · error · TypeError

Tokenizer returned unexpected types.

Error message

Tokenizer returned unexpected types.

What it means

After calling the tokenizer, _encode_prompt verifies that input_ids and attention_mask are torch.Tensors. If the tokenizer returns lists, None, or other container types instead, it raises a TypeError, since downstream code indexes into tensor shapes and moves them to device.

Source

Thrown at invokeai/app/invocations/anima_text_encoder.py:166

                raise TypeError(f"Expected PreTrainedTokenizerBase for tokenizer, got {type(tokenizer).__name__}.")

            context.util.signal_progress("Running Qwen3 0.6B text encoder")

            # Anima uses base Qwen3 (not instruct) — tokenize directly, no chat template.
            # A safety cap is applied to prevent GPU OOM on extremely long prompts.
            text_inputs = tokenizer(
                prompt,
                padding=False,
                truncation=True,
                max_length=QWEN3_MAX_SEQ_LEN,
                return_attention_mask=True,
                return_tensors="pt",
            )

            text_input_ids = text_inputs.input_ids
            attention_mask = text_inputs.attention_mask
            if not isinstance(text_input_ids, torch.Tensor) or not isinstance(attention_mask, torch.Tensor):
                raise TypeError("Tokenizer returned unexpected types.")

            if text_input_ids.shape[-1] == QWEN3_MAX_SEQ_LEN:
                logger.warning(
                    f"Prompt was truncated to {QWEN3_MAX_SEQ_LEN} tokens. "
                    "Consider shortening the prompt for best results."
                )

            # Ensure at least 1 token (empty prompts produce 0 tokens with padding=False)
            if text_input_ids.shape[-1] == 0:
                pad_id = tokenizer.pad_token_id if tokenizer.pad_token_id is not None else tokenizer.eos_token_id
                text_input_ids = torch.tensor([[pad_id]])
                attention_mask = torch.tensor([[1]])

            # Get last hidden state from Qwen3 (final layer output)
            prompt_mask = attention_mask.to(device).bool()
            outputs = text_encoder(
                text_input_ids.to(device),
                attention_mask=prompt_mask,

View on GitHub (pinned to 0b6a024f2f)

Solutions

Use the standard transformers tokenizer for the model, not a custom subclass or wrapper.
Explicitly convert with torch.tensor(text_inputs.input_ids) before proceeding if a custom tokenizer must be used.
Ensure the transformers version matches the installed InvokeAI requirements.

Example fix

// before
text_inputs = tokenizer(prompt, return_tensors="pt")  # custom tokenizer returns lists
// after
text_inputs = tokenizer(prompt, return_tensors="pt")
if not isinstance(text_inputs.input_ids, torch.Tensor):
    text_inputs.input_ids = torch.tensor(text_inputs.input_ids)

Defensive patterns

Strategy: type-guard

Validate before calling

import torch
out = tokenizer(prompt, return_tensors="pt")
assert isinstance(out.input_ids, torch.Tensor) and isinstance(out.attention_mask, torch.Tensor)

Type guard

import torch
def tokenizer_output_ok(out) -> bool:
    return isinstance(getattr(out, "input_ids", None), torch.Tensor) and isinstance(getattr(out, "attention_mask", None), torch.Tensor)

Try / catch

try:
    result = invocation.invoke(context)
except TypeError as e:
    if "Tokenizer returned unexpected types" in str(e):
        swap_to_stock_transformers_tokenizer()
    else:
        raise

Prevention

Use the stock transformers tokenizer; avoid subclasses/wrappers that change return types.
Pin the transformers version InvokeAI expects.
If mocking tokenizers in tests, return BatchEncoding with tensor fields.

When it happens

Trigger: During invoke → _encode_prompt, when the Qwen3 tokenizer call returns a BatchEncoding whose input_ids or attention_mask are not torch tensors — e.g. return_tensors='pt' was not honored, a custom/subclassed tokenizer returned lists, or a stub tokenizer returned None.

Common situations: A tokenizer monkey-patched or wrapped by another integration; transformers version where the tokenizer returns nested lists for edge inputs; testing with a mock tokenizer that doesn't emulate tensor conversion.

Related errors

AI-assisted analysis of invoke-ai/InvokeAI@0b6a024f2f (2026-08-29). Data as JSON: /api/errors/7658adeebe5d998d. Report an issue: GitHub.