invoke-ai/InvokeAI · error · TypeError

Expected PreTrainedTokenizerBase for Gemma tokenizer, got {t

Error message

Expected PreTrainedTokenizerBase for Gemma tokenizer, got {type(gemma_tokenizer).__name__}.

What it means

The Gemma tokenizer must be an instance of transformers PreTrainedTokenizerBase for PiD decode caption encoding. A different object indicates corrupted or missing tokenizer files, or a wrong model record for the tokenizer.

Source

Thrown at invokeai/app/invocations/flux2_pid_decode.py:175

                if config is not None and hasattr(config, "scaling_factor"):
                    scaling_factor = float(config.scaling_factor)
                    shift_factor = float(getattr(config, "shift_factor", None) or 0.0)
                else:
                    scaling_factor = float(getattr(vae, "scale_factor", scaling_factor))
                    shift_factor = float(getattr(vae, "shift_factor", shift_factor))
            del vae_info
            TorchDevice.empty_cache()

        # 3) Encode caption with Gemma-2.
        gemma_text_encoder_info = context.models.load(self.gemma2_encoder.text_encoder)
        gemma_tokenizer_info = context.models.load(self.gemma2_encoder.tokenizer)
        with ExitStack() as stack:
            (_, gemma_encoder) = stack.enter_context(gemma_text_encoder_info.model_on_device())
            (_, gemma_tokenizer) = stack.enter_context(gemma_tokenizer_info.model_on_device())
            if not isinstance(gemma_encoder, PreTrainedModel):
                raise TypeError(f"Expected PreTrainedModel for Gemma encoder, got {type(gemma_encoder).__name__}.")
            if not isinstance(gemma_tokenizer, PreTrainedTokenizerBase):
                raise TypeError(
                    f"Expected PreTrainedTokenizerBase for Gemma tokenizer, got {type(gemma_tokenizer).__name__}."
                )

            # Encode on the encoder's intended compute device. compute_device honours cpu_only and is
            # stable under partial loading — the first parameter may be offloaded to CPU while later
            # modules load on CUDA, so inferring the device from the first parameter could place caption
            # inputs on the wrong device.
            device = gemma_text_encoder_info.compute_device
            encode_dtype = TorchDevice.choose_bfloat16_safe_dtype(device)
            context.util.signal_progress("Encoding caption with Gemma-2")
            caption_embs, caption_mask = encode_caption_for_pid(
                [self.prompt],
                tokenizer=gemma_tokenizer,
                encoder=gemma_encoder,
                device=device,
                dtype=encode_dtype,
            )
            caption_embs = caption_embs.detach().to("cpu")

View on GitHub (pinned to 0b6a024f2f)

Solutions

1. Re-download the Gemma tokenizer files (tokenizer.json, tokenizer_config.json, special_tokens_map.json)
2. Verify gemma2_encoder.tokenizer references the correct tokenizer record
3. Update transformers to a compatible version
4. Re-import the Gemma model bundle if records are stale

Example fix

// before: tokenizer dir incomplete
/models/gemma2/ (config.json, model.safetensors)
// after
/models/gemma2/ (+ tokenizer.json, tokenizer_config.json, special_tokens_map.json)

info = context.models.load(gemma2_encoder.tokenizer)
if not isinstance(info.model, PreTrainedTokenizerBase):
    raise TypeError(f'Gemma tokenizer invalid: {type(info.model).__name__}')

from transformers import PreTrainedTokenizerBase

def is_gemma_tokenizer(obj) -> bool:
    return isinstance(obj, PreTrainedTokenizerBase)

try:
    result = pid_decode.invoke(context)
except TypeError as e:
    if 'Gemma tokenizer' in str(e):
        redownload_tokenizer_files(gemma2_encoder.tokenizer)
    raise

When it happens

Trigger: gemma_tokenizer_info.model_on_device() returns a non-tokenizer object in invoke; tokenizer files absent from the Gemma model directory or the gemma2_encoder.tokenizer field points elsewhere.

Common situations: Tokenizer files skipped during download; tokenizer record pointing to a different model's tokenizer; custom tokenizer classes not deriving from PreTrainedTokenizerBase; transformers version drift.

Related errors

• Expected PreTrainedTokenizerBase for tokenizer, got {type(to
• Expected PreTrainedModel for text encoder, got {type(text_en
• Expected ModelPatchRaw for LoRA '{lora.lora.key}', got {type
• Expected PreTrainedModel for Gemma encoder, got {type(gemma_
• Expected PidNet for PiD decoder, got {type(pid_net).__name__

AI-assisted analysis of invoke-ai/InvokeAI@0b6a024f2f (2026-08-29). Data as JSON: /api/errors/c01bf892a7ff6c6f. Report an issue: GitHub.