{"record":{"id":"28e1b872783ff038","repo":"unslothai/unsloth","slug":"mlx-vlm-s-registered-renderer-returned-an-empty-pr","errorCode":null,"errorMessage":"mlx-vlm's registered renderer returned an empty prompt.","messagePattern":"mlx-vlm's registered renderer returned an empty prompt\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"studio/backend/core/inference/mlx_inference.py","lineNumber":173,"sourceCode":"        # later capability and export logic observe a value it never published.\n        config = dict(config) if isinstance(config, dict) else dict(config.__dict__)\n        config[\"model_type\"] = canonical\n\n    # Recovery path: sweeps the caller's original list rather than reusing a copy (#7066).\n    swept = neutralize_control_markup_in_messages(messages, None, markup_for_tokenizer(processor))\n    partial = trailing_assistant_text(swept) if continue_final_message else None\n    rendered = prompt_utils.apply_chat_template(\n        processor,\n        config,\n        swept[:-1] if partial else swept,\n        add_generation_prompt = True,\n        num_images = num_images,\n        num_audios = num_audios,\n    )\n    if isinstance(rendered, str) and rendered.strip():\n        # A prefilled open \"<think>\" would resume the answer inside the reasoning block.\n        return f\"{strip_open_reasoning_prefill(rendered)}{partial}\" if partial else rendered\n    raise RuntimeError(\"mlx-vlm's registered renderer returned an empty prompt.\")\n\n\n# Rate the chat route decodes uploads to; mlx-vlm does not resample arrays.\n_AUDIO_INPUT_SAMPLE_RATE = 16000\n_AUDIO_PROBE_MESSAGES = [{\"role\": \"user\", \"content\": \"audio\"}]\n# Same turn with and without an image part, so a diff isolates the image marker.\n_IMAGE_PROBE_MESSAGES = [\n    {\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"hi\"}]}\n]\n_TEXT_PROBE_MESSAGES = [{\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": \"hi\"}]}]\n\n\ndef _classify_mlx_audio_type(\n    model,\n    processor,\n    is_vision,\n    config_audio_type = None,\n):","sourceCodeStart":155,"sourceCodeEnd":191,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/core/inference/mlx_inference.py#L155-L191","documentation":"Raised during prompt construction for MLX vision-language inference when mlx-vlm's registered chat-template renderer returns an empty or whitespace-only string. The backend calls prompt_utils.apply_chat_template with the (possibly assistant-prefilled) message list and num_images/num_audios counts; a non-empty string is required because an empty prompt cannot be tokenized for generation. Empty output usually means the model's chat template silently dropped all content — commonly because the declared multimodal part counts do not match the actual message content.","triggerScenarios":"Calling VLM generation where apply_chat_template returns '' — e.g. num_images=0 while messages contain an image part (or vice versa), a chat template that renders nothing when it receives an unexpected content type, or continuing a final message whose swept history becomes empty after the partial-text sweep.","commonSituations":"Mismatch between num_images/num_audios kwargs and the actual multimodal parts in messages; a VLM repo whose chat_template is audio-only or image-only receiving the other modality; edge case where swept[:-1] leaves no messages when continue_final_message=True and history has one message.","solutions":["Check that num_images and num_audios exactly match the image/audio parts present in messages.","If continue_final_message=True, ensure history contains at least one message besides the trailing assistant text being continued.","Test the repo's chat template directly (processor.apply_chat_template on a minimal multimodal message) to confirm the template supports the modality you are sending; if not, use a model that does."],"exampleFix":"# before\nprompt = apply_chat_template(processor, config, messages,\n    add_generation_prompt=True, num_images=0, num_audios=0)  # but messages contain an image\n\n# after\nnum_images = sum(1 for m in messages for p in as_parts(m) if p.get('type') == 'image')\nprompt = apply_chat_template(processor, config, messages,\n    add_generation_prompt=True, num_images=num_images, num_audios=0)","handlingStrategy":"validation","validationCode":"num_images = sum(1 for m in messages for p in (m.get('content') or [] if isinstance(m.get('content'), list) else []) if isinstance(p, dict) and p.get('type') == 'image')\nif is_vision and num_images == 0 and any_image_parts(messages):\n    raise ValueError('num_images mismatch')","typeGuard":null,"tryCatchPattern":"try:\n    rendered = render_prompt(processor, config, messages, ...)\nexcept RuntimeError as e:\n    if 'empty prompt' in str(e):\n        # retry once with recomputed part counts, no prefill\n        rendered = render_prompt(processor, config, messages,\n            add_generation_prompt=True, num_images=count_images(messages), num_audios=count_audios(messages))\n    else:\n        raise","preventionTips":["Derive num_images/num_audios from the message content instead of trusting caller-supplied counts.","Smoke-test a repo's chat template with one image + one text message at load time; disable VLM route if it renders empty.","When continue_final_message=True, assert the swept history is non-empty before slicing."],"tags":["mlx","vlm","chat-template","prompting"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}