sgl-project/sglang · error · ValueError

Unknown multimodal item type: {type(item)}

Error message

Unknown multimodal item type: {type(item)}

What it means

process_and_combine_mm_data sorts incoming items by modality; an item whose modality is not IMAGE, AUDIO, or VIDEO falls through to this ValueError. It guards against new/unknown modality enum values reaching the combine stage.

Source

Thrown at python/sglang/srt/multimodal/processors/base_processor.py:1685

            input_ids = tokenizer(
                base_output.input_text,
                return_tensors="pt",
                add_special_tokens=True,
            ).input_ids.flatten()
            return [], input_ids, {}

        dict_items, raw_images, raw_audios, raw_videos = [], [], [], []
        for modality, item in all_loaded_data:
            if isinstance(item, dict):
                dict_items.append((modality, item))
            elif modality == Modality.IMAGE:
                raw_images.append(item)
            elif modality == Modality.AUDIO:
                raw_audios.append(item)
            elif modality == Modality.VIDEO:
                raw_videos.append(item)
            else:
                raise ValueError(f"Unknown multimodal item type: {type(item)}")
        # Process items and get input_ids
        all_collected_items: list[MultimodalDataItem] = []
        input_ids = None
        # Handle raw items (need processing)
        if raw_images or raw_audios or raw_videos:
            if processor_override is not None:
                kwargs["processor"] = processor
            collected_items, input_ids, ret = self._process_and_collect_mm_items(
                input_text=base_output.input_text,
                images=raw_images,
                audios=raw_audios,
                videos=raw_videos,
                **kwargs,
            )
            all_collected_items = collected_items

            # When SGLANG_MM_AVOID_RETOKENIZE is on, keep the user's exact tokens to avoid retokenize drift.
            # Drift happens when Retokenization is not identity: Decode(X) => String => Re-tokenize => Y, X != Y.

View on GitHub (pinned to 0132848349)

Solutions

1. Filter items to Modality.IMAGE/AUDIO/VIDEO before calling, or route other modalities through their dedicated path
2. If adding a new modality, extend this router (raw_* handling) in the same change
3. Check for version mismatch between sglang components after upgrades

Example fix

// before
items = all_mm_items  # contains Modality.EMBEDDING
await processor.process_and_combine_mm_data(items, ...)
// after
items = [i for i in all_mm_items if i.modality in (Modality.IMAGE, Modality.AUDIO, Modality.VIDEO)]
await processor.process_and_combine_mm_data(items, ...)

from sglang.srt.multimodal import Modality
SUPPORTED = {Modality.IMAGE, Modality.AUDIO, Modality.VIDEO}
def routable(item):
    return getattr(item, 'modality', None) in SUPPORTED

When it happens

Trigger: Passing an item whose Modality enum value is something other than IMAGE/AUDIO/VIDEO (e.g. a future or custom modality) into process_and_combine_mm_data.

Common situations: Extending Modality with a new kind without updating this router; mixed-content dicts misclassified upstream; version skew between components enumerating modalities differently.

Related errors

• Unsupported image type: {type(image)}
• When using multiple prompts with multiple input images, prov
• {key}.position_ids is required
• {key}.{field} is required
• You have to specify input_ids

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/0896554bf03f648e. Report an issue: GitHub.