invoke-ai/InvokeAI · error · TypeError

Expected PreTrainedTokenizerBase for tokenizer, got {type(to

Error message

Expected PreTrainedTokenizerBase for tokenizer, got {type(tokenizer).__name__}.

What it means

Alongside the encoder check, _encode_prompt requires the tokenizer to be a transformers PreTrainedTokenizerBase. A non-tokenizer object (or wrong tokenizer class/artifact) loaded for the model key triggers this TypeError, which names the actual type received.

Source

Thrown at invokeai/app/invocations/anima_text_encoder.py:148

            # have temporarily offloaded all weights to RAM, which would wrongly run the whole encode on the CPU (see
            # #9373). Qwen3 is fully autocast-capable, so nothing pins it to the compute device otherwise.
            device = text_encoder_info.compute_device

            # Apply LoRA models to the text encoder
            lora_dtype = TorchDevice.choose_anima_inference_dtype(device)
            exit_stack.enter_context(
                LayerPatcher.apply_smart_model_patches(
                    model=text_encoder,
                    patches=self._lora_iterator(context),
                    prefix=ANIMA_LORA_QWEN3_PREFIX,
                    dtype=lora_dtype,
                )
            )

            if not isinstance(text_encoder, PreTrainedModel):
                raise TypeError(f"Expected PreTrainedModel for text encoder, got {type(text_encoder).__name__}.")
            if not isinstance(tokenizer, PreTrainedTokenizerBase):
                raise TypeError(f"Expected PreTrainedTokenizerBase for tokenizer, got {type(tokenizer).__name__}.")

            context.util.signal_progress("Running Qwen3 0.6B text encoder")

            # Anima uses base Qwen3 (not instruct) — tokenize directly, no chat template.
            # A safety cap is applied to prevent GPU OOM on extremely long prompts.
            text_inputs = tokenizer(
                prompt,
                padding=False,
                truncation=True,
                max_length=QWEN3_MAX_SEQ_LEN,
                return_attention_mask=True,
                return_tensors="pt",
            )

            text_input_ids = text_inputs.input_ids
            attention_mask = text_inputs.attention_mask
            if not isinstance(text_input_ids, torch.Tensor) or not isinstance(attention_mask, torch.Tensor):
                raise TypeError("Tokenizer returned unexpected types.")

View on GitHub (pinned to 0b6a024f2f)

Solutions

Re-download/re-import the model so the tokenizer directory contains valid tokenizer_config.json/tokenizer.json.
Verify the tokenizer record in the model manager points to the Qwen3 tokenizer files, not another component.
Update the transformers library and reload the tokenizer.

Defensive patterns

Strategy: type-guard

Validate before calling

from transformers import PreTrainedTokenizerBase
info = context.models.load(tokenizer_key)
if not isinstance(info.model, PreTrainedTokenizerBase):
    print(f"Tokenizer is {type(info.model).__name__}, re-import required")

Type guard

from transformers import PreTrainedTokenizerBase
def is_valid_tokenizer(obj) -> bool:
    return isinstance(obj, PreTrainedTokenizerBase)

Try / catch

try:
    result = invocation.invoke(context)
except TypeError as e:
    if "Expected PreTrainedTokenizerBase for tokenizer" in str(e):
        reimport_tokenizer_files()
    else:
        raise

Prevention

Ensure complete model downloads include tokenizer_config.json/tokenizer.json.
Point model-manager tokenizer records at the correct tokenizer directory.
Avoid manually replacing tokenizer files with custom classes.

When it happens

Trigger: During invoke → _encode_prompt, when isinstance(tokenizer, PreTrainedTokenizerBase) fails because the loaded tokenizer artifact is not a HF tokenizer — wrong file in the tokenizer slot, corrupted tokenizer.json, or a config pointing at a non-tokenizer object.

Common situations: Tokenizer files missing from the model directory after an incomplete download; tokenizer.json replaced by a custom class; converting models with tooling that drops tokenizer artifacts.

Related errors

AI-assisted analysis of invoke-ai/InvokeAI@0b6a024f2f (2026-08-29). Data as JSON: /api/errors/4d70fa9c0fa33aa8. Report an issue: GitHub.