unslothai/unsloth · error · RuntimeError

mlx-vlm's registered renderer returned an empty prompt.

Error message

mlx-vlm's registered renderer returned an empty prompt.

What it means

Raised during prompt construction for MLX vision-language inference when mlx-vlm's registered chat-template renderer returns an empty or whitespace-only string. The backend calls prompt_utils.apply_chat_template with the (possibly assistant-prefilled) message list and num_images/num_audios counts; a non-empty string is required because an empty prompt cannot be tokenized for generation. Empty output usually means the model's chat template silently dropped all content — commonly because the declared multimodal part counts do not match the actual message content.

Source

Thrown at studio/backend/core/inference/mlx_inference.py:173

        # later capability and export logic observe a value it never published.
        config = dict(config) if isinstance(config, dict) else dict(config.__dict__)
        config["model_type"] = canonical

    # Recovery path: sweeps the caller's original list rather than reusing a copy (#7066).
    swept = neutralize_control_markup_in_messages(messages, None, markup_for_tokenizer(processor))
    partial = trailing_assistant_text(swept) if continue_final_message else None
    rendered = prompt_utils.apply_chat_template(
        processor,
        config,
        swept[:-1] if partial else swept,
        add_generation_prompt = True,
        num_images = num_images,
        num_audios = num_audios,
    )
    if isinstance(rendered, str) and rendered.strip():
        # A prefilled open "<think>" would resume the answer inside the reasoning block.
        return f"{strip_open_reasoning_prefill(rendered)}{partial}" if partial else rendered
    raise RuntimeError("mlx-vlm's registered renderer returned an empty prompt.")


# Rate the chat route decodes uploads to; mlx-vlm does not resample arrays.
_AUDIO_INPUT_SAMPLE_RATE = 16000
_AUDIO_PROBE_MESSAGES = [{"role": "user", "content": "audio"}]
# Same turn with and without an image part, so a diff isolates the image marker.
_IMAGE_PROBE_MESSAGES = [
    {"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "hi"}]}
]
_TEXT_PROBE_MESSAGES = [{"role": "user", "content": [{"type": "text", "text": "hi"}]}]


def _classify_mlx_audio_type(
    model,
    processor,
    is_vision,
    config_audio_type = None,
):

View on GitHub (pinned to 203007d190)

Solutions

1. Check that num_images and num_audios exactly match the image/audio parts present in messages.
2. If continue_final_message=True, ensure history contains at least one message besides the trailing assistant text being continued.
3. Test the repo's chat template directly (processor.apply_chat_template on a minimal multimodal message) to confirm the template supports the modality you are sending; if not, use a model that does.

Example fix

# before
prompt = apply_chat_template(processor, config, messages,
    add_generation_prompt=True, num_images=0, num_audios=0)  # but messages contain an image

# after
num_images = sum(1 for m in messages for p in as_parts(m) if p.get('type') == 'image')
prompt = apply_chat_template(processor, config, messages,
    add_generation_prompt=True, num_images=num_images, num_audios=0)

num_images = sum(1 for m in messages for p in (m.get('content') or [] if isinstance(m.get('content'), list) else []) if isinstance(p, dict) and p.get('type') == 'image')
if is_vision and num_images == 0 and any_image_parts(messages):
    raise ValueError('num_images mismatch')

try:
    rendered = render_prompt(processor, config, messages, ...)
except RuntimeError as e:
    if 'empty prompt' in str(e):
        # retry once with recomputed part counts, no prefill
        rendered = render_prompt(processor, config, messages,
            add_generation_prompt=True, num_images=count_images(messages), num_audios=count_audios(messages))
    else:
        raise

When it happens

Trigger: Calling VLM generation where apply_chat_template returns '' — e.g. num_images=0 while messages contain an image part (or vice versa), a chat template that renders nothing when it receives an unexpected content type, or continuing a final message whose swept history becomes empty after the partial-text sweep.

Common situations: Mismatch between num_images/num_audios kwargs and the actual multimodal parts in messages; a VLM repo whose chat_template is audio-only or image-only receiving the other modality; edge case where swept[:-1] leaves no messages when continue_final_message=True and history has one message.

Related errors

• the template produced an empty prompt
• apply_chat_template returned None — tokenizer may be incompa
• apply_chat_template_for_generation: no attempt produced a re
• no attempt rendered the continuation prefix
• Model '{self.active_model_name}' has no chat_template set in

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/28e1b872783ff038. Report an issue: GitHub.