{"record":{"id":"66c08b16efc60891","repo":"zylon-ai/private-gpt","slug":"multimodal-input-provided-but-tokenizer-is-not-mul","errorCode":null,"errorMessage":"Multimodal input provided but tokenizer is not multimodal","messagePattern":"Multimodal input provided but tokenizer is not multimodal","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"private_gpt/components/llm/tokenizers/huggingface.py","lineNumber":203,"sourceCode":"        if not texts:\n            return []\n\n        batch_encoding = self._tokenizer(\n            texts,\n            add_special_tokens=False,\n        )\n        text_input_ids: list[int] = batch_encoding[\"input_ids\"]\n        return text_input_ids\n\n    def calculate_mm_input_ids(\n        self,\n        texts: TextLike | None = None,\n        images: ImageLike | None = None,\n        audios: AudioLike | None = None,\n    ) -> list[int]:\n        \"\"\"Estimate tokens for images and audio using processor.\"\"\"\n        if not self._processor:\n            raise ValueError(\n                \"Multimodal input provided but tokenizer is not multimodal\"\n            )\n\n        current_conversation: Any = self._tokenizer.apply_chat_template(\n            build_minimal_messages(images=images, audios=audios),\n            add_generation_prompt=False,\n            tokenize=True,\n            return_dict=True,\n            return_tensors=\"pt\",\n        )\n\n        total_input_ids = current_conversation[\"input_ids\"][0]\n        baseline_input_ids = self._empty_conversation[\"input_ids\"][0]\n        return [int(id) for id in total_input_ids if id not in baseline_input_ids]\n\n    def get_vocab(self) -> dict[str, int]:\n        vocab: dict[str, int] = self._tokenizer.get_vocab()\n        return vocab","sourceCodeStart":185,"sourceCodeEnd":221,"githubUrl":"https://github.com/zylon-ai/private-gpt/blob/4a030776a31a901ad80b1bf4d7faa2c1a367efbb/private_gpt/components/llm/tokenizers/huggingface.py#L185-L221","documentation":"calculate_mm_input_ids estimates tokens for images/audio by running the model's multimodal processor. The processor only exists when from_pretrained loaded an AutoProcessor that has a .tokenizer attribute; a plain text tokenizer leaves _processor None, so passing images/audio raises ValueError.","triggerScenarios":"Calling calculate_mm_input_ids(images=[...]) or (audios=[...]) on a HuggingFaceTokenizer built from a text-only model (e.g. Mistral-7B), or when AutoProcessor.from_pretrained returned a plain tokenizer instead of a ProcessorMixin.","commonSituations":"Sending image attachments to a text-only model in the ingestion/context path; misconfigured multimodal flag or model id pointing to the text checkpoint of a multimodal family; token estimation code assuming multimodal support unconditionally.","solutions":["Use a multimodal model id (one whose repo yields a ProcessorMixin, e.g. a VLM) so the tokenizer is built with a processor.","Guard the call site: check tokenizer.is_multimodal / _processor before sending images or audio.","If the request is really text-only, strip image/audio parts from the payload before token estimation."],"exampleFix":"# before\nids = tokenizer.calculate_mm_input_ids(texts=[t], images=[img])  # text-only model\n\n# after\nif images or audios:\n    assert tokenizer.is_multimodal, 'model/tokenizer does not accept images or audio'\nids = tokenizer.calculate_mm_input_ids(texts=[t], images=[img])","handlingStrategy":"type-guard","validationCode":"def accepts_multimodal(tokenizer) -> bool:\n    return bool(getattr(tokenizer, '_processor', None)) or bool(getattr(tokenizer, 'is_multimodal', False))","typeGuard":"def is_multimodal_tokenizer(t: object) -> bool:\n    return bool(getattr(t, '_processor', None))","tryCatchPattern":"try:\n    ids = tokenizer.calculate_mm_input_ids(texts=t, images=imgs)\nexcept ValueError as e:\n    if 'not multimodal' in str(e):\n        raise UnsupportedInput('model is text-only; remove image/audio parts') from e\n    raise","preventionTips":["Check tokenizer.is_multimodal before accepting attachments at the API boundary.","Bind multimodal acceptance to the deployed model family in config.","Return a 4xx (not 500) when clients send images to a text-only model."],"tags":["multimodal","tokenizer","validation"],"backgroundTag":null,"analyzedSha":"4a030776a31a901ad80b1bf4d7faa2c1a367efbb","analyzedAt":"2026-08-15T03:51:26.951Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}