{"record":{"id":"4d70fa9c0fa33aa8","repo":"invoke-ai/InvokeAI","slug":"expected-pretrainedtokenizerbase-for-tokenizer-go","errorCode":null,"errorMessage":"Expected PreTrainedTokenizerBase for tokenizer, got {type(tokenizer).__name__}.","messagePattern":"Expected PreTrainedTokenizerBase for tokenizer, got (.+?)\\.","errorType":"exception","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"invokeai/app/invocations/anima_text_encoder.py","lineNumber":148,"sourceCode":"            # have temporarily offloaded all weights to RAM, which would wrongly run the whole encode on the CPU (see\n            # #9373). Qwen3 is fully autocast-capable, so nothing pins it to the compute device otherwise.\n            device = text_encoder_info.compute_device\n\n            # Apply LoRA models to the text encoder\n            lora_dtype = TorchDevice.choose_anima_inference_dtype(device)\n            exit_stack.enter_context(\n                LayerPatcher.apply_smart_model_patches(\n                    model=text_encoder,\n                    patches=self._lora_iterator(context),\n                    prefix=ANIMA_LORA_QWEN3_PREFIX,\n                    dtype=lora_dtype,\n                )\n            )\n\n            if not isinstance(text_encoder, PreTrainedModel):\n                raise TypeError(f\"Expected PreTrainedModel for text encoder, got {type(text_encoder).__name__}.\")\n            if not isinstance(tokenizer, PreTrainedTokenizerBase):\n                raise TypeError(f\"Expected PreTrainedTokenizerBase for tokenizer, got {type(tokenizer).__name__}.\")\n\n            context.util.signal_progress(\"Running Qwen3 0.6B text encoder\")\n\n            # Anima uses base Qwen3 (not instruct) — tokenize directly, no chat template.\n            # A safety cap is applied to prevent GPU OOM on extremely long prompts.\n            text_inputs = tokenizer(\n                prompt,\n                padding=False,\n                truncation=True,\n                max_length=QWEN3_MAX_SEQ_LEN,\n                return_attention_mask=True,\n                return_tensors=\"pt\",\n            )\n\n            text_input_ids = text_inputs.input_ids\n            attention_mask = text_inputs.attention_mask\n            if not isinstance(text_input_ids, torch.Tensor) or not isinstance(attention_mask, torch.Tensor):\n                raise TypeError(\"Tokenizer returned unexpected types.\")","sourceCodeStart":130,"sourceCodeEnd":166,"githubUrl":"https://github.com/invoke-ai/InvokeAI/blob/0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06/invokeai/app/invocations/anima_text_encoder.py#L130-L166","documentation":"Alongside the encoder check, _encode_prompt requires the tokenizer to be a transformers PreTrainedTokenizerBase. A non-tokenizer object (or wrong tokenizer class/artifact) loaded for the model key triggers this TypeError, which names the actual type received.","triggerScenarios":"During invoke → _encode_prompt, when isinstance(tokenizer, PreTrainedTokenizerBase) fails because the loaded tokenizer artifact is not a HF tokenizer — wrong file in the tokenizer slot, corrupted tokenizer.json, or a config pointing at a non-tokenizer object.","commonSituations":"Tokenizer files missing from the model directory after an incomplete download; tokenizer.json replaced by a custom class; converting models with tooling that drops tokenizer artifacts.","solutions":["Re-download/re-import the model so the tokenizer directory contains valid tokenizer_config.json/tokenizer.json.","Verify the tokenizer record in the model manager points to the Qwen3 tokenizer files, not another component.","Update the transformers library and reload the tokenizer."],"exampleFix":null,"handlingStrategy":"type-guard","validationCode":"from transformers import PreTrainedTokenizerBase\ninfo = context.models.load(tokenizer_key)\nif not isinstance(info.model, PreTrainedTokenizerBase):\n    print(f\"Tokenizer is {type(info.model).__name__}, re-import required\")","typeGuard":"from transformers import PreTrainedTokenizerBase\ndef is_valid_tokenizer(obj) -> bool:\n    return isinstance(obj, PreTrainedTokenizerBase)","tryCatchPattern":"try:\n    result = invocation.invoke(context)\nexcept TypeError as e:\n    if \"Expected PreTrainedTokenizerBase for tokenizer\" in str(e):\n        reimport_tokenizer_files()\n    else:\n        raise","preventionTips":["Ensure complete model downloads include tokenizer_config.json/tokenizer.json.","Point model-manager tokenizer records at the correct tokenizer directory.","Avoid manually replacing tokenizer files with custom classes."],"tags":["type-mismatch","tokenizer","qwen3","invokeai"],"backgroundTag":"unexpected-model-type","analyzedSha":"0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06","analyzedAt":"2026-08-29T04:46:49.967Z","schemaVersion":2},"datasetVersion":"2026-08-29T07:17:48.351Z"}