sgl-project/sglang · error · ValueError

Found more '{modality_token}' placeholders in input prompt t

Error message

Found more '{modality_token}' placeholders in input prompt than actual multimodal data items.

What it means

When building a multimodal prompt, SGLang counts how many modality placeholder tokens (e.g. <image>) already exist in the user's text prompt and compares against the number of multimodal data items supplied. If the prompt contains more placeholders than actual images/videos/audios provided, rendering would leave dangling placeholders, so it raises this ValueError.

Source

Thrown at python/sglang/srt/parser/conversation.py:593

    return convs


# Models in which system adds modality tokens at prompt start automatically
# when media inputs exceed modality tokens in prompt (e.g. 3 images but 2 <image> tokens)
_MODELS_REQUIRING_MODALITY_SUPPLEMENT = {"deepseek-vl2"}


# adapted from https://github.com/vllm-project/vllm/blob/5124f5bf51b83e6f344c1bc6652e8c4d81313b34/vllm/entrypoints/chat_utils.py#L856
def _get_full_multimodal_text_prompt(
    modality_token: str, modality_count: int, text_prompt: str
) -> str:
    """Combine multimodal prompts for a multimodal language model."""

    # For any existing placeholder in the text prompt, we leave it as is
    left: int = modality_count - text_prompt.count(modality_token)
    if left < 0:
        raise ValueError(
            f"Found more '{modality_token}' placeholders in input prompt than "
            "actual multimodal data items."
        )

    # NOTE: For now we always add missing modality_token at the front of
    # the prompt. This may change to be customizable in the future.
    return "\n".join([modality_token] * left + [text_prompt])


def generate_chat_conv(
    request: ChatCompletionRequest, template_name: str
) -> Conversation:
    conv = chat_templates[template_name].copy()
    conv = Conversation(
        name=conv.name,
        system_template=conv.system_template,
        system_message=conv.system_message,
        roles=conv.roles,

View on GitHub (pinned to 0132848349)

Solutions

1. Make the number of modality_token occurrences in the prompt equal or fewer than the supplied multimodal items (missing ones are auto-prepended)
2. Remove hardcoded <image>/<video>/<audio> tokens from the prompt and let SGLang insert them
3. Pass additional image/video/audio data items to match the placeholder count

Example fix

# before
prompt = "Compare <image> and <image>"
image_data = [img1]  # only one image
# after
prompt = "Compare <image> and <image>"
image_data = [img1, img2]  # or drop tokens from the prompt

tok = conv.image_token
count = sum(p.count(tok) for m in messages if isinstance(m.get("content"), str) for p in [m["content"]])
assert count <= len(image_data), f"{count} placeholders vs {len(image_data)} items"

When it happens

Trigger: Calling generate_chat_conv with a message like 'Compare <image> and <image>' but supplying only one image in request.image_data; or a prompt that hardcodes the image token multiple times while modalities list has fewer entries.

Common situations: Users write template prompts with multiple <image> tokens for multi-image comparison but attach fewer images; or upstream code reuses a prompt template that embeds the token while separately passing image data of a different length.

Related errors

• Prompt cannot be empty
• Z-Image text embeddings must have shape [seq, dim] or [batch
• Cosmos3 action prompt must be a string or non-empty list
• f"Unsupported patch_size type: {type(patch_size)}"
• f"Expected camera embedding shape [B, C, F, H, W], got {tupl

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f1df44b0b532f95d. Report an issue: GitHub.