sgl-project/sglang · error · ValueError

[internvl] Cannot process raw images/videos with pre-tokeniz

Error message

[internvl] Cannot process raw images/videos with pre-tokenized input_ids. Provide multimodal data in 'processor_output' or 'precomputed_embedding' format, or use a text prompt instead. (raw images dropped: {raw_img_dropped}, raw videos dropped: {raw_vid_dropped})

What it means

InternVL's processor needs the raw prompt string to do dynamic tiling and expand image/video placeholder tokens. When a request arrives pre-tokenized (input_ids) together with raw PIL/bytes images or videos, those raw items are filtered out and the processor raises rather than silently dropping them. Pre-extracted features ('processor_output') or 'precomputed_embedding' entries are the supported way to send multimodal data alongside input_ids.

Source

Thrown at python/sglang/srt/multimodal/processors/internvl.py:299

            prompt = ""
        else:
            user_input_ids = None
            prompt = input_text or ""

        # When the prompt is empty (user provided input_ids directly),
        # load_mm_data can't match multimodal tokens to data items.
        # Build BaseMultiModalProcessorOutput directly from the dict items.
        if not prompt and (image_data or video_data):
            images = [d for d in (image_data or []) if isinstance(d, dict)]
            videos = [d for d in (video_data or []) if isinstance(d, dict)]

            # Raise if raw (non-dict) images/videos were silently filtered out.
            # InternVL cannot process raw images without a text prompt because
            # dynamic tiling and placeholder expansion require the prompt string.
            raw_img_dropped = len(image_data or []) - len(images)
            raw_vid_dropped = len(video_data or []) - len(videos)
            if raw_img_dropped > 0 or raw_vid_dropped > 0:
                raise ValueError(
                    f"[internvl] Cannot process raw images/videos with pre-tokenized "
                    f"input_ids. Provide multimodal data in 'processor_output' or "
                    f"'precomputed_embedding' format, or use a text prompt instead. "
                    f"(raw images dropped: {raw_img_dropped}, "
                    f"raw videos dropped: {raw_vid_dropped})"
                )

            base_output = BaseMultiModalProcessorOutput(
                input_text=prompt,
                images=images,
                videos=videos,
            )
        else:
            base_output = await self.load_mm_data(
                prompt=prompt,
                image_data=image_data,
                video_data=video_data,
                multimodal_tokens=self.mm_tokens,

View on GitHub (pinned to 0132848349)

Solutions

  1. Send a text prompt (input_text) instead of input_ids when supplying raw images/videos
  2. Or pre-extract features and pass them as 'processor_output'/'precomputed_embedding' dicts in image_data
  3. Keep the request shape consistent: raw media + text, or input_ids + precomputed tensors

Example fix

# before
req.input_ids = tokenizer(prompt)['input_ids']
out = await proc.process_mm_data_async(None, image_data=[pil_img], request_obj=req)
# after
req.input_text = prompt  # let the processor tokenize + expand placeholders
out = await proc.process_mm_data_async(None, image_data=[pil_img], request_obj=req)
Defensive patterns

Strategy: type-guard

Validate before calling

def is_raw_media(x): return not isinstance(x, dict)
if getattr(req,'input_ids',None) is not None and any(is_raw_media(m) for m in (image_data or [])+(video_data or [])):
    raise UserInputError('send text prompt with raw media, or precomputed dicts with input_ids')

Type guard

def is_precomputed(x) -> bool:
    return isinstance(x, dict) and ('processor_output' in x or 'precomputed_embedding' in x)

Try / catch

try:
    out = await proc.process_mm_data_async(None, image_data, req)
except ValueError as e:
    if 'pre-tokenized' in str(e):
        del req.input_ids; req.input_text = prompt  # retry with text
        out = await proc.process_mm_data_async(None, image_data, req)
    else: raise

Prevention

When it happens

Trigger: Calling process_mm_data_async with request_obj.input_ids set and image_data containing PIL images or video paths; mixing pre-tokenized prompts with raw media in offline batch inference.

Common situations: Batch pipelines that pre-tokenize for caching then attach raw frames; upgrading a pipeline that previously (incorrectly) tolerated the silent drop; sending OpenAI-style image_url content that resolves to raw bytes with input_ids requests.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/cc58c6cdc95691c3. Report an issue: GitHub.