{"record":{"id":"cc58c6cdc95691c3","repo":"sgl-project/sglang","slug":"internvl-cannot-process-raw-images-videos-with-p","errorCode":null,"errorMessage":"[internvl] Cannot process raw images/videos with pre-tokenized input_ids. Provide multimodal data in 'processor_output' or 'precomputed_embedding' format, or use a text prompt instead. (raw images dropped: {raw_img_dropped}, raw videos dropped: {raw_vid_dropped})","messagePattern":"\\[internvl\\] Cannot process raw images/videos with pre-tokenized input_ids\\. Provide multimodal data in 'processor_output' or 'precomputed_embedding' format, or use a text prompt instead\\. \\(raw images dropped: (.+?), raw videos dropped: (.+?)\\)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/multimodal/processors/internvl.py","lineNumber":299,"sourceCode":"            prompt = \"\"\n        else:\n            user_input_ids = None\n            prompt = input_text or \"\"\n\n        # When the prompt is empty (user provided input_ids directly),\n        # load_mm_data can't match multimodal tokens to data items.\n        # Build BaseMultiModalProcessorOutput directly from the dict items.\n        if not prompt and (image_data or video_data):\n            images = [d for d in (image_data or []) if isinstance(d, dict)]\n            videos = [d for d in (video_data or []) if isinstance(d, dict)]\n\n            # Raise if raw (non-dict) images/videos were silently filtered out.\n            # InternVL cannot process raw images without a text prompt because\n            # dynamic tiling and placeholder expansion require the prompt string.\n            raw_img_dropped = len(image_data or []) - len(images)\n            raw_vid_dropped = len(video_data or []) - len(videos)\n            if raw_img_dropped > 0 or raw_vid_dropped > 0:\n                raise ValueError(\n                    f\"[internvl] Cannot process raw images/videos with pre-tokenized \"\n                    f\"input_ids. Provide multimodal data in 'processor_output' or \"\n                    f\"'precomputed_embedding' format, or use a text prompt instead. \"\n                    f\"(raw images dropped: {raw_img_dropped}, \"\n                    f\"raw videos dropped: {raw_vid_dropped})\"\n                )\n\n            base_output = BaseMultiModalProcessorOutput(\n                input_text=prompt,\n                images=images,\n                videos=videos,\n            )\n        else:\n            base_output = await self.load_mm_data(\n                prompt=prompt,\n                image_data=image_data,\n                video_data=video_data,\n                multimodal_tokens=self.mm_tokens,","sourceCodeStart":281,"sourceCodeEnd":317,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/multimodal/processors/internvl.py#L281-L317","documentation":"InternVL's processor needs the raw prompt string to do dynamic tiling and expand image/video placeholder tokens. When a request arrives pre-tokenized (input_ids) together with raw PIL/bytes images or videos, those raw items are filtered out and the processor raises rather than silently dropping them. Pre-extracted features ('processor_output') or 'precomputed_embedding' entries are the supported way to send multimodal data alongside input_ids.","triggerScenarios":"Calling process_mm_data_async with request_obj.input_ids set and image_data containing PIL images or video paths; mixing pre-tokenized prompts with raw media in offline batch inference.","commonSituations":"Batch pipelines that pre-tokenize for caching then attach raw frames; upgrading a pipeline that previously (incorrectly) tolerated the silent drop; sending OpenAI-style image_url content that resolves to raw bytes with input_ids requests.","solutions":["Send a text prompt (input_text) instead of input_ids when supplying raw images/videos","Or pre-extract features and pass them as 'processor_output'/'precomputed_embedding' dicts in image_data","Keep the request shape consistent: raw media + text, or input_ids + precomputed tensors"],"exampleFix":"# before\nreq.input_ids = tokenizer(prompt)['input_ids']\nout = await proc.process_mm_data_async(None, image_data=[pil_img], request_obj=req)\n# after\nreq.input_text = prompt  # let the processor tokenize + expand placeholders\nout = await proc.process_mm_data_async(None, image_data=[pil_img], request_obj=req)","handlingStrategy":"type-guard","validationCode":"def is_raw_media(x): return not isinstance(x, dict)\nif getattr(req,'input_ids',None) is not None and any(is_raw_media(m) for m in (image_data or [])+(video_data or [])):\n    raise UserInputError('send text prompt with raw media, or precomputed dicts with input_ids')","typeGuard":"def is_precomputed(x) -> bool:\n    return isinstance(x, dict) and ('processor_output' in x or 'precomputed_embedding' in x)","tryCatchPattern":"try:\n    out = await proc.process_mm_data_async(None, image_data, req)\nexcept ValueError as e:\n    if 'pre-tokenized' in str(e):\n        del req.input_ids; req.input_text = prompt  # retry with text\n        out = await proc.process_mm_data_async(None, image_data, req)\n    else: raise","preventionTips":["Pick one request shape: raw media+text OR input_ids+precomputed dicts","Cache precomputed processor_output when pre-tokenizing prompts"],"tags":["multimodal","internvl","pre-tokenized-input","dynamic-tiling"],"backgroundTag":"raw-media-with-pretokenized-input","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}