unslothai/unsloth · error · RuntimeError

No model loaded

Error message

No model loaded

What it means

Straightforward guard at the top of the MLX generation entry point: if self._model is None the backend has no loaded model, so streaming cannot proceed and RuntimeError('No model loaded') is raised immediately. This happens when generation is invoked before a successful load() or after the model was unloaded/crashed.

Source

Thrown at studio/backend/core/inference/mlx_inference.py:1438

        image = None,
        temperature = 0.7,
        top_p = 0.9,
        top_k = 40,
        min_p = 0.0,
        max_new_tokens = 256,
        repetition_penalty = 1.0,
        cancel_event = None,
        # Reasoning / tool kwargs, rendered via apply_chat_template_for_generation (transformers parity).
        tools = None,
        enable_thinking = None,
        reasoning_effort = None,
        preserve_thinking = None,
        continue_final_message = False,
        presence_penalty = 0.0,
        _adapter_state = None,
    ) -> Generator[str, None, None]:
        if self._model is None:
            raise RuntimeError("No model loaded")

        # Reset so a failed run cannot surface stale stats.
        self.last_generation_stats = None

        full_messages = []
        if system_prompt:
            full_messages.append({"role": "system", "content": system_prompt})
        full_messages.extend(messages)

        # Inject image into the last user message for VLM
        if self._is_vlm and image is not None:
            for msg in reversed(full_messages):
                if msg.get("role") == "user":
                    content = msg.get("content", "")
                    if isinstance(content, str):
                        msg["content"] = [
                            {"type": "image"},
                            {"type": "text", "text": content},

View on GitHub (pinned to 203007d190)

Solutions

1. Load a model first and confirm it succeeded (check active_model_name / load result) before issuing generation requests.
2. If loading previously failed, address that root-cause error — 'No model loaded' is only the downstream symptom.
3. Gate the API route: return 503 until the backend reports a loaded model.

Example fix

# before
backend = MLXInferenceBackend(...)
out = backend.stream_response(messages)  # RuntimeError: No model loaded

# after
backend.load('mlx-community/Llama-3.1-8B-Instruct-4bit')
out = backend.stream_response(messages)

if backend._model is None or not backend.active_model_name:
    raise ServiceUnavailable('model not loaded; call load() first')

def has_loaded_model(backend) -> bool:
    return getattr(backend, '_model', None) is not None

try:
    for chunk in backend.stream_response(messages):
        yield chunk
except RuntimeError as e:
    if str(e) == 'No model loaded':
        yield error_event('model_not_loaded', retry_after_load=True)
    else:
        raise

When it happens

Trigger: Calling stream/generate on a fresh MLXInferenceBackend instance without load(); calling after unload(); calling after a load that raised (the error path left _model None).

Common situations: API route hit before the model finished loading (race at service startup); load failed with an earlier error and the caller ignored it; client reconnecting after a model-swap sequence.

Related errors

• No diffusion model is loaded.
• Unsloth MLX: use_adapter must be None, True, False, or a str
• Unsloth MLX: cannot disable adapter layers without their bas
• the template produced an empty prompt
• apply_chat_template returned None — tokenizer may be incompa

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/a807c1fa64bc84c7. Report an issue: GitHub.