{"record":{"id":"0c2aadf994992378","repo":"invoke-ai/InvokeAI","slug":"expected-pretrainedtokenizerbase-for-tokenizer-go-0c2aad","errorCode":null,"errorMessage":"Expected PreTrainedTokenizerBase for tokenizer, got {type(tokenizer).__name__}. The Qwen3 tokenizer may be corrupted or incompatible.","messagePattern":"Expected PreTrainedTokenizerBase for tokenizer, got (.+?)\\. The Qwen3 tokenizer may be corrupted or incompatible\\.","errorType":"exception","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"invokeai/app/invocations/flux2_klein_text_encoder.py","lineNumber":141,"sourceCode":"        exit_stack.enter_context(\n            LayerPatcher.apply_smart_model_patches(\n                model=text_encoder,\n                patches=self._lora_iterator(context),\n                prefix=FLUX_LORA_T5_PREFIX,\n                dtype=lora_dtype,\n                cached_weights=cached_weights,\n            )\n        )\n\n        context.util.signal_progress(\"Running Qwen3 text encoder (Klein)\")\n\n        if not isinstance(text_encoder, PreTrainedModel):\n            raise TypeError(\n                f\"Expected PreTrainedModel for text encoder, got {type(text_encoder).__name__}. \"\n                \"The Qwen3 encoder model may be corrupted or incompatible.\"\n            )\n        if not isinstance(tokenizer, PreTrainedTokenizerBase):\n            raise TypeError(\n                f\"Expected PreTrainedTokenizerBase for tokenizer, got {type(tokenizer).__name__}. \"\n                \"The Qwen3 tokenizer may be corrupted or incompatible.\"\n            )\n\n        messages = [{\"role\": \"user\", \"content\": prompt}]\n\n        text: str = tokenizer.apply_chat_template(  # type: ignore[assignment]\n            messages,\n            tokenize=False,\n            add_generation_prompt=True,\n            enable_thinking=False,\n        )\n\n        inputs = tokenizer(\n            text,\n            return_tensors=\"pt\",\n            padding=\"max_length\",\n            truncation=True,","sourceCodeStart":123,"sourceCodeEnd":159,"githubUrl":"https://github.com/invoke-ai/InvokeAI/blob/0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06/invokeai/app/invocations/flux2_klein_text_encoder.py#L123-L159","documentation":"This invocation requires the Qwen3 tokenizer to be an instance of transformers PreTrainedTokenizerBase. Any other object means the tokenizer loaded from the model manager is corrupted, incomplete, or not a compatible tokenizer. The downstream chat-template encoding requires the standard transformers tokenizer API.","triggerScenarios":"context.models.load() on self.qwen3_encoder.tokenizer yields a non-tokenizer object in _encode_prompt; tokenizer.json/tokenizer_config.json missing or truncated; wrong directory referenced in the encoder node config.","commonSituations":"Incomplete model downloads where tokenizer files were skipped; users copying only weight files; a tokenizer replaced by a custom/non-transformers class; version drift between saved model records and installed transformers.","solutions":["Re-download the Qwen3 tokenizer files (tokenizer.json, tokenizer_config.json, vocab files)","Verify the tokenizer model record points at the correct Qwen3 tokenizer directory","Check transformers/diffusers versions are current and compatible","Delete and re-import the FLUX.2 Klein model bundle in the model manager"],"exampleFix":"// before: tokenizer dir missing tokenizer.json\n/models/Qwen3-encoder/  (config.json, model.safetensors only)\n// after: complete files\n/models/Qwen3-encoder/  (config.json, model.safetensors, tokenizer.json, tokenizer_config.json)","handlingStrategy":"type-guard","validationCode":"info = context.models.load(qwen3_encoder.tokenizer)\nif not isinstance(info.model, PreTrainedTokenizerBase):\n    raise TypeError(f'Qwen3 tokenizer invalid: {type(info.model).__name__}')","typeGuard":"from transformers import PreTrainedTokenizerBase\n\ndef is_valid_tokenizer(obj) -> bool:\n    return isinstance(obj, PreTrainedTokenizerBase)","tryCatchPattern":"try:\n    result = klein_encoder.invoke(context)\nexcept TypeError as e:\n    if 'PreTrainedTokenizerBase for tokenizer' in str(e):\n        reimport_tokenizer_files(qwen3_encoder.tokenizer)\n    raise","preventionTips":["Ensure tokenizer.json and tokenizer_config.json are present after download","Do not copy only weight files when moving model folders","Pin transformers versions known to work with the model"],"tags":["tokenizer","model-loading","type-check"],"backgroundTag":"model-type-mismatch","analyzedSha":"0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06","analyzedAt":"2026-08-29T04:46:49.967Z","schemaVersion":2},"datasetVersion":"2026-08-29T07:17:48.351Z"}