sgl-project/sglang · error · NotImplementedError

gRPC encode only supports IMAGE modality, got: {non_image}

Error message

gRPC encode only supports IMAGE modality, got: {non_image}

What it means

Raised in the gRPC encode path when any item in mm_data carries a modality other than IMAGE. The gRPC encoder interface currently only transports image payloads, so requests containing video or audio items are rejected with NotImplementedError.

Source

Thrown at python/sglang/srt/disaggregation/encoder/receiver.py:2579

        embedding_port,
        endpoint_encode,
        num_items_assigned=None,
        encode_urls=None,
    ):
        if not mm_data:
            return

        effective_urls = encode_urls if encode_urls is not None else self.encode_urls

        # gRPC currently only supports image; flatten new dict formats to simple lists
        if mm_data and isinstance(mm_data[0], dict):
            non_image = [
                item.get("modality")
                for item in mm_data
                if item.get("modality") != Modality.IMAGE
            ]
            if non_image:
                raise NotImplementedError(
                    f"gRPC encode only supports IMAGE modality, got: {non_image}"
                )
            img_data = [item.get("url") for item in mm_data]
        else:
            img_data = mm_data
        if isinstance(num_items_assigned, dict):
            num_items_assigned = list(num_items_assigned.values())[0]

        encode_requests = []
        if num_items_assigned is None:
            encode_idx = list(range(len(effective_urls)))
            random.shuffle(encode_idx)
            num_items_assigned = [
                (idx + len(img_data)) // len(effective_urls) for idx in encode_idx
            ]
        num_parts = sum(1 for x in num_items_assigned if x != 0)
        cum_num_items = 0
        cum_idx = 0

View on GitHub (pinned to 0132848349)

Solutions

1. Send image-only requests to gRPC encoders, or strip video/audio items before encoding.
2. Switch to HTTP encoder URLs / receiver mode which supports the broader modality set.
3. Check upstream request construction: video_data or audio_data being non-empty triggers this.

Example fix

# before
req.video_data = [video_item]  # with grpc encoder urls
# after
req.video_data = None  # or use http:// encoder urls for video workloads

def is_image_only(mm_data: list) -> bool:
    return all(item.get("modality") == Modality.IMAGE for item in mm_data)

if transport_mode == "grpc":
    assert is_image_only(mm_data), "gRPC encode path accepts IMAGE items only"

def is_image_request(req) -> bool:
    return not (req.video_data or req.audio_data)

try:
    result = await receiver.encode(...)
except NotImplementedError as e:
    if "IMAGE modality" in str(e):
        raise HTTPException(400, "video/audio not supported over gRPC encoders; use http mode") from e
    raise

When it happens

Trigger: Calling encode on the gRPC receiver with mm_data entries whose "modality" is Modality.VIDEO or Modality.AUDIO; e.g. a multimodal request that includes a video_url or audio input while using grpc:// encoder URLs.

Common situations: Pointing a vision-language-plus-audio workload (e.g. Qwen-Audio/Video style requests) at a gRPC encoder deployment; the receiver auto-flattening video_data/audio_data into mm_data which then fails the image-only check.

Related errors

• Invalid modality: {modality}
• {error_msg}
• Encoder produced {mm_embedding.shape[0]} tokens, but preproc
• Encoder produced {mm_embedding.shape[0]} tokens, but preproc
• MM inputs where only some items are precomputed.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/099c27ffb04efd08. Report an issue: GitHub.