zylon-ai/private-gpt · error · ValueError

Multimodal input provided but tokenizer is not multimodal

Error message

Multimodal input provided but tokenizer is not multimodal

What it means

calculate_mm_input_ids estimates tokens for images/audio by running the model's multimodal processor. The processor only exists when from_pretrained loaded an AutoProcessor that has a .tokenizer attribute; a plain text tokenizer leaves _processor None, so passing images/audio raises ValueError.

Source

Thrown at private_gpt/components/llm/tokenizers/huggingface.py:203

        if not texts:
            return []

        batch_encoding = self._tokenizer(
            texts,
            add_special_tokens=False,
        )
        text_input_ids: list[int] = batch_encoding["input_ids"]
        return text_input_ids

    def calculate_mm_input_ids(
        self,
        texts: TextLike | None = None,
        images: ImageLike | None = None,
        audios: AudioLike | None = None,
    ) -> list[int]:
        """Estimate tokens for images and audio using processor."""
        if not self._processor:
            raise ValueError(
                "Multimodal input provided but tokenizer is not multimodal"
            )

        current_conversation: Any = self._tokenizer.apply_chat_template(
            build_minimal_messages(images=images, audios=audios),
            add_generation_prompt=False,
            tokenize=True,
            return_dict=True,
            return_tensors="pt",
        )

        total_input_ids = current_conversation["input_ids"][0]
        baseline_input_ids = self._empty_conversation["input_ids"][0]
        return [int(id) for id in total_input_ids if id not in baseline_input_ids]

    def get_vocab(self) -> dict[str, int]:
        vocab: dict[str, int] = self._tokenizer.get_vocab()
        return vocab

View on GitHub (pinned to 4a030776a3)

Solutions

1. Use a multimodal model id (one whose repo yields a ProcessorMixin, e.g. a VLM) so the tokenizer is built with a processor.
2. Guard the call site: check tokenizer.is_multimodal / _processor before sending images or audio.
3. If the request is really text-only, strip image/audio parts from the payload before token estimation.

Example fix

# before
ids = tokenizer.calculate_mm_input_ids(texts=[t], images=[img])  # text-only model

# after
if images or audios:
    assert tokenizer.is_multimodal, 'model/tokenizer does not accept images or audio'
ids = tokenizer.calculate_mm_input_ids(texts=[t], images=[img])

def accepts_multimodal(tokenizer) -> bool:
    return bool(getattr(tokenizer, '_processor', None)) or bool(getattr(tokenizer, 'is_multimodal', False))

def is_multimodal_tokenizer(t: object) -> bool:
    return bool(getattr(t, '_processor', None))

try:
    ids = tokenizer.calculate_mm_input_ids(texts=t, images=imgs)
except ValueError as e:
    if 'not multimodal' in str(e):
        raise UnsupportedInput('model is text-only; remove image/audio parts') from e
    raise

When it happens

Trigger: Calling calculate_mm_input_ids(images=[...]) or (audios=[...]) on a HuggingFaceTokenizer built from a text-only model (e.g. Mistral-7B), or when AutoProcessor.from_pretrained returned a plain tokenizer instead of a ProcessorMixin.

Common situations: Sending image attachments to a text-only model in the ingestion/context path; misconfigured multimodal flag or model id pointing to the text checkpoint of a multimodal family; token estimation code assuming multimodal support unconditionally.

Related errors

• RemoteTokenizeTokenizer only supports text tokenization
• Tokenizer is required and must support apply_chat_template:
• RemoteTokenizeTokenizer is not available with the given conf
• TikTokenTokenizer only supports text token counting
• tokenizer_fn must be provided.

AI-assisted analysis of zylon-ai/private-gpt@4a030776a3 (2026-08-15). Data as JSON: /api/errors/66c08b16efc60891. Report an issue: GitHub.