invoke-ai/InvokeAI · error · TypeError

Expected PreTrainedTokenizerBase for tokenizer, got {type(to

Error message

Expected PreTrainedTokenizerBase for tokenizer, got {type(tokenizer).__name__}. The Qwen3 tokenizer may be corrupted or incompatible.

What it means

This invocation requires the Qwen3 tokenizer to be an instance of transformers PreTrainedTokenizerBase. Any other object means the tokenizer loaded from the model manager is corrupted, incomplete, or not a compatible tokenizer. The downstream chat-template encoding requires the standard transformers tokenizer API.

Source

Thrown at invokeai/app/invocations/flux2_klein_text_encoder.py:141

        exit_stack.enter_context(
            LayerPatcher.apply_smart_model_patches(
                model=text_encoder,
                patches=self._lora_iterator(context),
                prefix=FLUX_LORA_T5_PREFIX,
                dtype=lora_dtype,
                cached_weights=cached_weights,
            )
        )

        context.util.signal_progress("Running Qwen3 text encoder (Klein)")

        if not isinstance(text_encoder, PreTrainedModel):
            raise TypeError(
                f"Expected PreTrainedModel for text encoder, got {type(text_encoder).__name__}. "
                "The Qwen3 encoder model may be corrupted or incompatible."
            )
        if not isinstance(tokenizer, PreTrainedTokenizerBase):
            raise TypeError(
                f"Expected PreTrainedTokenizerBase for tokenizer, got {type(tokenizer).__name__}. "
                "The Qwen3 tokenizer may be corrupted or incompatible."
            )

        messages = [{"role": "user", "content": prompt}]

        text: str = tokenizer.apply_chat_template(  # type: ignore[assignment]
            messages,
            tokenize=False,
            add_generation_prompt=True,
            enable_thinking=False,
        )

        inputs = tokenizer(
            text,
            return_tensors="pt",
            padding="max_length",
            truncation=True,

View on GitHub (pinned to 0b6a024f2f)

Solutions

1. Re-download the Qwen3 tokenizer files (tokenizer.json, tokenizer_config.json, vocab files)
2. Verify the tokenizer model record points at the correct Qwen3 tokenizer directory
3. Check transformers/diffusers versions are current and compatible
4. Delete and re-import the FLUX.2 Klein model bundle in the model manager

Example fix

// before: tokenizer dir missing tokenizer.json
/models/Qwen3-encoder/  (config.json, model.safetensors only)
// after: complete files
/models/Qwen3-encoder/  (config.json, model.safetensors, tokenizer.json, tokenizer_config.json)

info = context.models.load(qwen3_encoder.tokenizer)
if not isinstance(info.model, PreTrainedTokenizerBase):
    raise TypeError(f'Qwen3 tokenizer invalid: {type(info.model).__name__}')

from transformers import PreTrainedTokenizerBase

def is_valid_tokenizer(obj) -> bool:
    return isinstance(obj, PreTrainedTokenizerBase)

try:
    result = klein_encoder.invoke(context)
except TypeError as e:
    if 'PreTrainedTokenizerBase for tokenizer' in str(e):
        reimport_tokenizer_files(qwen3_encoder.tokenizer)
    raise

When it happens

Trigger: context.models.load() on self.qwen3_encoder.tokenizer yields a non-tokenizer object in _encode_prompt; tokenizer.json/tokenizer_config.json missing or truncated; wrong directory referenced in the encoder node config.

Common situations: Incomplete model downloads where tokenizer files were skipped; users copying only weight files; a tokenizer replaced by a custom/non-transformers class; version drift between saved model records and installed transformers.

Related errors

• Expected PreTrainedTokenizerBase for Gemma tokenizer, got {t
• Expected PreTrainedModel for text encoder, got {type(text_en
• Expected ModelPatchRaw for LoRA '{lora.lora.key}', got {type
• Expected PreTrainedModel for Gemma encoder, got {type(gemma_
• Expected PidNet for PiD decoder, got {type(pid_net).__name__

AI-assisted analysis of invoke-ai/InvokeAI@0b6a024f2f (2026-08-29). Data as JSON: /api/errors/0c2aadf994992378. Report an issue: GitHub.