invoke-ai/InvokeAI · error · TypeError

Expected PreTrainedTokenizerBase for tokenizer, got {type(to

Error message

Expected PreTrainedTokenizerBase for tokenizer, got {type(tokenizer).__name__}. The Qwen3 tokenizer may be corrupted or incompatible.

What it means

_encode_prompt also validates the Qwen3 tokenizer with isinstance(tokenizer, PreTrainedTokenizerBase) before applying the chat template. A tokenizer that is not a real HF tokenizer raises a TypeError noting the Qwen3 tokenizer may be corrupted or incompatible. This ensures prompt formatting and encoding use the standard tokenizer API.

Source

Thrown at invokeai/app/invocations/z_image_text_encoder.py:112

            lora_dtype = TorchDevice.choose_bfloat16_safe_dtype(device)
            exit_stack.enter_context(
                LayerPatcher.apply_smart_model_patches(
                    model=text_encoder,
                    patches=self._lora_iterator(context),
                    prefix=Z_IMAGE_LORA_QWEN3_PREFIX,
                    dtype=lora_dtype,
                    cached_weights=cached_weights,
                )
            )

            context.util.signal_progress("Running Qwen3 text encoder")
            if not isinstance(text_encoder, PreTrainedModel):
                raise TypeError(
                    f"Expected PreTrainedModel for text encoder, got {type(text_encoder).__name__}. "
                    "The Qwen3 encoder model may be corrupted or incompatible."
                )
            if not isinstance(tokenizer, PreTrainedTokenizerBase):
                raise TypeError(
                    f"Expected PreTrainedTokenizerBase for tokenizer, got {type(tokenizer).__name__}. "
                    "The Qwen3 tokenizer may be corrupted or incompatible."
                )

            # Apply chat template similar to diffusers ZImagePipeline
            # The chat template formats the prompt for the Qwen3 model
            try:
                prompt_formatted = tokenizer.apply_chat_template(
                    [{"role": "user", "content": prompt}],
                    tokenize=False,
                    add_generation_prompt=True,
                    enable_thinking=True,
                )
            except (AttributeError, TypeError) as e:
                # Fallback if tokenizer doesn't support apply_chat_template or enable_thinking
                context.logger.warning(f"Chat template failed ({e}), using raw prompt.")
                prompt_formatted = prompt

View on GitHub (pinned to 0b6a024f2f)

Solutions

Re-download the Qwen3 tokenizer files (tokenizer.json, tokenizer_config.json, vocab files).
Re-import the Z-Image model so the tokenizer submodel is properly registered as an HF tokenizer.
Update transformers/InvokeAI to compatible versions.
Verify the tokenizer submodel reference in the invocation matches the Qwen3 encoder.

Defensive patterns

Strategy: type-guard

Validate before calling

with tokenizer_info.model_on_device() as (_, tok):
    if not isinstance(tok, PreTrainedTokenizerBase):
        fail_fast(tok)

Type guard

def is_hf_tokenizer(obj) -> bool:
    from transformers import PreTrainedTokenizerBase
    return isinstance(obj, PreTrainedTokenizerBase)

Try / catch

try:
    encode(context)
except TypeError as e:
    if "Expected PreTrainedTokenizerBase for tokenizer" in str(e):
        repair_qwen3_tokenizer_files()
    else:
        raise

Prevention

Keep the complete Qwen3 tokenizer file set next to the encoder.
Avoid manually replacing tokenizer files with partial copies.
Confirm submodel bindings after model re-imports.

When it happens

Trigger: Z-Image text encoding where the tokenizer submodel loaded via model_on_device fails isinstance(tokenizer, PreTrainedTokenizerBase).

Common situations: Missing tokenizer files in the Qwen3 tokenizer directory (tokenizer.json etc.); corrupted download; tokenizer submodel pointing to a wrong model; transformers version incompatibility.

Related errors

AI-assisted analysis of invoke-ai/InvokeAI@0b6a024f2f (2026-08-29). Data as JSON: /api/errors/f8cb6701e9736e66. Report an issue: GitHub.