jamiepine/voicebox · error · HTTPException

LLM generation failed

Error message

LLM generation failed

What it means

Generic 500 from POST /llm/generate. The route wraps backend.generate() in a bare except Exception; any failure — model not loaded, OOM, tokenizer error, CUDA/MPS fault, malformed prompt — is logged via logger.exception server-side and rewritten to the opaque string 'LLM generation failed' to avoid leaking filesystem paths or stack frames to the client. The original exception is chained via 'from e'.

Source

Thrown at backend/routes/llm.py:80

                    detail="Each example must be a [user, assistant] pair",
                )
        examples = [(pair[0], pair[1]) for pair in request.examples]

    try:
        text = await backend.generate(
            prompt=request.prompt,
            system=request.system,
            max_tokens=request.max_tokens,
            temperature=request.temperature,
            model_size=model_size,
            examples=examples,
        )
        return models.LLMGenerateResponse(text=text, model_size=model_size)
    except Exception as e:
        # The backend exception text can include filesystem paths and stack
        # frames — log it server-side and hand the client a generic message.
        logger.exception("LLM generate failed")
        raise HTTPException(status_code=500, detail="LLM generation failed") from e

View on GitHub (pinned to 51f49dea19)

Solutions

1. Check the server logs — logger.exception wrote the real traceback under 'LLM generate failed'. The client only sees the sanitized message.
2. Retry with a smaller model_size (0.6B) and/or lower max_tokens to rule out memory pressure.
3. Shorten the prompt and examples; very long inputs can exceed the model's context window.
4. Ensure no concurrent unload/migrate is touching the LLM backend while generating.
5. If the error is persistent, restart the backend and confirm GPU/CPU setup with GET /health before retrying.

Example fix

// before
fetch('/llm/generate', {method:'POST', body: JSON.stringify({prompt: hugeText, model_size:'4B', max_tokens:4096})})
// after
fetch('/llm/generate', {method:'POST', body: JSON.stringify({prompt: hugeText.slice(0,4000), model_size:'0.6B', max_tokens:512})})

// Pre-flight: confirm the LLM is loaded and a smaller size is available
const status = await (await fetch('/models/status')).json();
const llm = status.models.find(m => m.engine === 'qwen_llm' && m.loaded);
if (!llm) {
  // load via /models/download, then poll /models/progress until complete
  throw new Error('LLM not loaded — start download first');
}

let lastErr;
for (const size of ['0.6B','1.7B','4B']) {
  try {
    const r = await fetch('/llm/generate', {method:'POST', body: JSON.stringify({...body, model_size: size})});
    if (r.ok) return await r.json();
    if (r.status !== 500) { lastErr = await r.json(); break; }
    lastErr = await r.json().catch(() => ({}));
  } catch (e) { lastErr = e; }
}
throw new Error('LLM generation failed: ' + (lastErr?.detail ?? 'unknown'));

When it happens

Trigger: Calling /llm/generate when the LLM backend raised during generate(): VRAM exhaustion on the 4B model, tokenizer crash on a prompt exceeding the context window, model weights freed by a concurrent unload, torch/mlx runtime error, or a generate() implementation bug.

Common situations: Low-memory machine loading the 4B size; concurrent /models/{name}/unload freeing weights mid-request; prompt near 50000-char field cap blowing the context window; CUDA driver/library version mismatch; antivirus or OOM killer terminating the worker after weights loaded.

Related errors

• LLM produced empty output; nothing to speak.
• Unknown LLM engine: {engine}. Supported: {list(LLM_ENGINES.k
• Invalid LLM size '{model_size}'. Must be one of: {sorted(val
• Each example must be a [user, assistant] pair
• {e}

AI-assisted analysis of jamiepine/voicebox@51f49dea19 (2026-08-12). Data as JSON: /api/errors/ea6d8d847021cb6a. Report an issue: GitHub.