{"record":{"id":"c01bf892a7ff6c6f","repo":"invoke-ai/InvokeAI","slug":"expected-pretrainedtokenizerbase-for-gemma-tokeniz","errorCode":null,"errorMessage":"Expected PreTrainedTokenizerBase for Gemma tokenizer, got {type(gemma_tokenizer).__name__}.","messagePattern":"Expected PreTrainedTokenizerBase for Gemma tokenizer, got (.+?)\\.","errorType":"exception","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"invokeai/app/invocations/flux2_pid_decode.py","lineNumber":175,"sourceCode":"                if config is not None and hasattr(config, \"scaling_factor\"):\n                    scaling_factor = float(config.scaling_factor)\n                    shift_factor = float(getattr(config, \"shift_factor\", None) or 0.0)\n                else:\n                    scaling_factor = float(getattr(vae, \"scale_factor\", scaling_factor))\n                    shift_factor = float(getattr(vae, \"shift_factor\", shift_factor))\n            del vae_info\n            TorchDevice.empty_cache()\n\n        # 3) Encode caption with Gemma-2.\n        gemma_text_encoder_info = context.models.load(self.gemma2_encoder.text_encoder)\n        gemma_tokenizer_info = context.models.load(self.gemma2_encoder.tokenizer)\n        with ExitStack() as stack:\n            (_, gemma_encoder) = stack.enter_context(gemma_text_encoder_info.model_on_device())\n            (_, gemma_tokenizer) = stack.enter_context(gemma_tokenizer_info.model_on_device())\n            if not isinstance(gemma_encoder, PreTrainedModel):\n                raise TypeError(f\"Expected PreTrainedModel for Gemma encoder, got {type(gemma_encoder).__name__}.\")\n            if not isinstance(gemma_tokenizer, PreTrainedTokenizerBase):\n                raise TypeError(\n                    f\"Expected PreTrainedTokenizerBase for Gemma tokenizer, got {type(gemma_tokenizer).__name__}.\"\n                )\n\n            # Encode on the encoder's intended compute device. compute_device honours cpu_only and is\n            # stable under partial loading — the first parameter may be offloaded to CPU while later\n            # modules load on CUDA, so inferring the device from the first parameter could place caption\n            # inputs on the wrong device.\n            device = gemma_text_encoder_info.compute_device\n            encode_dtype = TorchDevice.choose_bfloat16_safe_dtype(device)\n            context.util.signal_progress(\"Encoding caption with Gemma-2\")\n            caption_embs, caption_mask = encode_caption_for_pid(\n                [self.prompt],\n                tokenizer=gemma_tokenizer,\n                encoder=gemma_encoder,\n                device=device,\n                dtype=encode_dtype,\n            )\n            caption_embs = caption_embs.detach().to(\"cpu\")","sourceCodeStart":157,"sourceCodeEnd":193,"githubUrl":"https://github.com/invoke-ai/InvokeAI/blob/0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06/invokeai/app/invocations/flux2_pid_decode.py#L157-L193","documentation":"The Gemma tokenizer must be an instance of transformers PreTrainedTokenizerBase for PiD decode caption encoding. A different object indicates corrupted or missing tokenizer files, or a wrong model record for the tokenizer.","triggerScenarios":"gemma_tokenizer_info.model_on_device() returns a non-tokenizer object in invoke; tokenizer files absent from the Gemma model directory or the gemma2_encoder.tokenizer field points elsewhere.","commonSituations":"Tokenizer files skipped during download; tokenizer record pointing to a different model's tokenizer; custom tokenizer classes not deriving from PreTrainedTokenizerBase; transformers version drift.","solutions":["Re-download the Gemma tokenizer files (tokenizer.json, tokenizer_config.json, special_tokens_map.json)","Verify gemma2_encoder.tokenizer references the correct tokenizer record","Update transformers to a compatible version","Re-import the Gemma model bundle if records are stale"],"exampleFix":"// before: tokenizer dir incomplete\n/models/gemma2/ (config.json, model.safetensors)\n// after\n/models/gemma2/ (+ tokenizer.json, tokenizer_config.json, special_tokens_map.json)","handlingStrategy":"type-guard","validationCode":"info = context.models.load(gemma2_encoder.tokenizer)\nif not isinstance(info.model, PreTrainedTokenizerBase):\n    raise TypeError(f'Gemma tokenizer invalid: {type(info.model).__name__}')","typeGuard":"from transformers import PreTrainedTokenizerBase\n\ndef is_gemma_tokenizer(obj) -> bool:\n    return isinstance(obj, PreTrainedTokenizerBase)","tryCatchPattern":"try:\n    result = pid_decode.invoke(context)\nexcept TypeError as e:\n    if 'Gemma tokenizer' in str(e):\n        redownload_tokenizer_files(gemma2_encoder.tokenizer)\n    raise","preventionTips":["Confirm tokenizer files exist in the Gemma model folder","Do not mix other models' tokenizers into Gemma records","Re-import the bundle after partial downloads"],"tags":["tokenizer","model-loading","type-check"],"backgroundTag":"model-type-mismatch","analyzedSha":"0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06","analyzedAt":"2026-08-29T04:46:49.967Z","schemaVersion":2},"datasetVersion":"2026-08-29T07:17:48.351Z"}