{"record":{"id":"f8cb6701e9736e66","repo":"invoke-ai/InvokeAI","slug":"expected-pretrainedtokenizerbase-for-tokenizer-go-f8cb67","errorCode":null,"errorMessage":"Expected PreTrainedTokenizerBase for tokenizer, got {type(tokenizer).__name__}. The Qwen3 tokenizer may be corrupted or incompatible.","messagePattern":"Expected PreTrainedTokenizerBase for tokenizer, got (.+?)\\. The Qwen3 tokenizer may be corrupted or incompatible\\.","errorType":"exception","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"invokeai/app/invocations/z_image_text_encoder.py","lineNumber":112,"sourceCode":"            lora_dtype = TorchDevice.choose_bfloat16_safe_dtype(device)\n            exit_stack.enter_context(\n                LayerPatcher.apply_smart_model_patches(\n                    model=text_encoder,\n                    patches=self._lora_iterator(context),\n                    prefix=Z_IMAGE_LORA_QWEN3_PREFIX,\n                    dtype=lora_dtype,\n                    cached_weights=cached_weights,\n                )\n            )\n\n            context.util.signal_progress(\"Running Qwen3 text encoder\")\n            if not isinstance(text_encoder, PreTrainedModel):\n                raise TypeError(\n                    f\"Expected PreTrainedModel for text encoder, got {type(text_encoder).__name__}. \"\n                    \"The Qwen3 encoder model may be corrupted or incompatible.\"\n                )\n            if not isinstance(tokenizer, PreTrainedTokenizerBase):\n                raise TypeError(\n                    f\"Expected PreTrainedTokenizerBase for tokenizer, got {type(tokenizer).__name__}. \"\n                    \"The Qwen3 tokenizer may be corrupted or incompatible.\"\n                )\n\n            # Apply chat template similar to diffusers ZImagePipeline\n            # The chat template formats the prompt for the Qwen3 model\n            try:\n                prompt_formatted = tokenizer.apply_chat_template(\n                    [{\"role\": \"user\", \"content\": prompt}],\n                    tokenize=False,\n                    add_generation_prompt=True,\n                    enable_thinking=True,\n                )\n            except (AttributeError, TypeError) as e:\n                # Fallback if tokenizer doesn't support apply_chat_template or enable_thinking\n                context.logger.warning(f\"Chat template failed ({e}), using raw prompt.\")\n                prompt_formatted = prompt\n","sourceCodeStart":94,"sourceCodeEnd":130,"githubUrl":"https://github.com/invoke-ai/InvokeAI/blob/0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06/invokeai/app/invocations/z_image_text_encoder.py#L94-L130","documentation":"_encode_prompt also validates the Qwen3 tokenizer with isinstance(tokenizer, PreTrainedTokenizerBase) before applying the chat template. A tokenizer that is not a real HF tokenizer raises a TypeError noting the Qwen3 tokenizer may be corrupted or incompatible. This ensures prompt formatting and encoding use the standard tokenizer API.","triggerScenarios":"Z-Image text encoding where the tokenizer submodel loaded via model_on_device fails isinstance(tokenizer, PreTrainedTokenizerBase).","commonSituations":"Missing tokenizer files in the Qwen3 tokenizer directory (tokenizer.json etc.); corrupted download; tokenizer submodel pointing to a wrong model; transformers version incompatibility.","solutions":["Re-download the Qwen3 tokenizer files (tokenizer.json, tokenizer_config.json, vocab files).","Re-import the Z-Image model so the tokenizer submodel is properly registered as an HF tokenizer.","Update transformers/InvokeAI to compatible versions.","Verify the tokenizer submodel reference in the invocation matches the Qwen3 encoder."],"exampleFix":null,"handlingStrategy":"type-guard","validationCode":"with tokenizer_info.model_on_device() as (_, tok):\n    if not isinstance(tok, PreTrainedTokenizerBase):\n        fail_fast(tok)","typeGuard":"def is_hf_tokenizer(obj) -> bool:\n    from transformers import PreTrainedTokenizerBase\n    return isinstance(obj, PreTrainedTokenizerBase)","tryCatchPattern":"try:\n    encode(context)\nexcept TypeError as e:\n    if \"Expected PreTrainedTokenizerBase for tokenizer\" in str(e):\n        repair_qwen3_tokenizer_files()\n    else:\n        raise","preventionTips":["Keep the complete Qwen3 tokenizer file set next to the encoder.","Avoid manually replacing tokenizer files with partial copies.","Confirm submodel bindings after model re-imports."],"tags":["type-check","tokenizer","qwen3","z-image"],"backgroundTag":"unexpected-model-type","analyzedSha":"0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06","analyzedAt":"2026-08-29T04:46:49.967Z","schemaVersion":2},"datasetVersion":"2026-08-29T07:17:48.351Z"}