{"record":{"id":"f1df44b0b532f95d","repo":"sgl-project/sglang","slug":"found-more-modality-token-placeholders-in-inpu","errorCode":null,"errorMessage":"Found more '{modality_token}' placeholders in input prompt than actual multimodal data items.","messagePattern":"Found more '(.+?)' placeholders in input prompt than actual multimodal data items\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/parser/conversation.py","lineNumber":593,"sourceCode":"\n    return convs\n\n\n# Models in which system adds modality tokens at prompt start automatically\n# when media inputs exceed modality tokens in prompt (e.g. 3 images but 2 <image> tokens)\n_MODELS_REQUIRING_MODALITY_SUPPLEMENT = {\"deepseek-vl2\"}\n\n\n# adapted from https://github.com/vllm-project/vllm/blob/5124f5bf51b83e6f344c1bc6652e8c4d81313b34/vllm/entrypoints/chat_utils.py#L856\ndef _get_full_multimodal_text_prompt(\n    modality_token: str, modality_count: int, text_prompt: str\n) -> str:\n    \"\"\"Combine multimodal prompts for a multimodal language model.\"\"\"\n\n    # For any existing placeholder in the text prompt, we leave it as is\n    left: int = modality_count - text_prompt.count(modality_token)\n    if left < 0:\n        raise ValueError(\n            f\"Found more '{modality_token}' placeholders in input prompt than \"\n            \"actual multimodal data items.\"\n        )\n\n    # NOTE: For now we always add missing modality_token at the front of\n    # the prompt. This may change to be customizable in the future.\n    return \"\\n\".join([modality_token] * left + [text_prompt])\n\n\ndef generate_chat_conv(\n    request: ChatCompletionRequest, template_name: str\n) -> Conversation:\n    conv = chat_templates[template_name].copy()\n    conv = Conversation(\n        name=conv.name,\n        system_template=conv.system_template,\n        system_message=conv.system_message,\n        roles=conv.roles,","sourceCodeStart":575,"sourceCodeEnd":611,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/parser/conversation.py#L575-L611","documentation":"When building a multimodal prompt, SGLang counts how many modality placeholder tokens (e.g. <image>) already exist in the user's text prompt and compares against the number of multimodal data items supplied. If the prompt contains more placeholders than actual images/videos/audios provided, rendering would leave dangling placeholders, so it raises this ValueError.","triggerScenarios":"Calling generate_chat_conv with a message like 'Compare <image> and <image>' but supplying only one image in request.image_data; or a prompt that hardcodes the image token multiple times while modalities list has fewer entries.","commonSituations":"Users write template prompts with multiple <image> tokens for multi-image comparison but attach fewer images; or upstream code reuses a prompt template that embeds the token while separately passing image data of a different length.","solutions":["Make the number of modality_token occurrences in the prompt equal or fewer than the supplied multimodal items (missing ones are auto-prepended)","Remove hardcoded <image>/<video>/<audio> tokens from the prompt and let SGLang insert them","Pass additional image/video/audio data items to match the placeholder count"],"exampleFix":"# before\nprompt = \"Compare <image> and <image>\"\nimage_data = [img1]  # only one image\n# after\nprompt = \"Compare <image> and <image>\"\nimage_data = [img1, img2]  # or drop tokens from the prompt","handlingStrategy":"validation","validationCode":"tok = conv.image_token\ncount = sum(p.count(tok) for m in messages if isinstance(m.get(\"content\"), str) for p in [m[\"content\"]])\nassert count <= len(image_data), f\"{count} placeholders vs {len(image_data)} items\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Never hardcode <image>/<video> tokens unless you control both sides","Let the server auto-prepend placeholders for missing tokens"],"tags":["multimodal","vision","prompt","validation"],"backgroundTag":"placeholder-count-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}