{"record":{"id":"ea6d8d847021cb6a","repo":"jamiepine/voicebox","slug":"llm-generation-failed","errorCode":null,"errorMessage":"LLM generation failed","messagePattern":"LLM generation failed","errorType":"http","errorClass":"HTTPException","httpStatus":500,"severity":"error","filePath":"backend/routes/llm.py","lineNumber":80,"sourceCode":"                    detail=\"Each example must be a [user, assistant] pair\",\n                )\n        examples = [(pair[0], pair[1]) for pair in request.examples]\n\n    try:\n        text = await backend.generate(\n            prompt=request.prompt,\n            system=request.system,\n            max_tokens=request.max_tokens,\n            temperature=request.temperature,\n            model_size=model_size,\n            examples=examples,\n        )\n        return models.LLMGenerateResponse(text=text, model_size=model_size)\n    except Exception as e:\n        # The backend exception text can include filesystem paths and stack\n        # frames — log it server-side and hand the client a generic message.\n        logger.exception(\"LLM generate failed\")\n        raise HTTPException(status_code=500, detail=\"LLM generation failed\") from e\n","sourceCodeStart":62,"sourceCodeEnd":81,"githubUrl":"https://github.com/jamiepine/voicebox/blob/51f49dea198384b4eb6087b72c17057c6eb1c1cd/backend/routes/llm.py#L62-L81","documentation":"Generic 500 from POST /llm/generate. The route wraps backend.generate() in a bare except Exception; any failure — model not loaded, OOM, tokenizer error, CUDA/MPS fault, malformed prompt — is logged via logger.exception server-side and rewritten to the opaque string 'LLM generation failed' to avoid leaking filesystem paths or stack frames to the client. The original exception is chained via 'from e'.","triggerScenarios":"Calling /llm/generate when the LLM backend raised during generate(): VRAM exhaustion on the 4B model, tokenizer crash on a prompt exceeding the context window, model weights freed by a concurrent unload, torch/mlx runtime error, or a generate() implementation bug.","commonSituations":"Low-memory machine loading the 4B size; concurrent /models/{name}/unload freeing weights mid-request; prompt near 50000-char field cap blowing the context window; CUDA driver/library version mismatch; antivirus or OOM killer terminating the worker after weights loaded.","solutions":["Check the server logs — logger.exception wrote the real traceback under 'LLM generate failed'. The client only sees the sanitized message.","Retry with a smaller model_size (0.6B) and/or lower max_tokens to rule out memory pressure.","Shorten the prompt and examples; very long inputs can exceed the model's context window.","Ensure no concurrent unload/migrate is touching the LLM backend while generating.","If the error is persistent, restart the backend and confirm GPU/CPU setup with GET /health before retrying."],"exampleFix":"// before\nfetch('/llm/generate', {method:'POST', body: JSON.stringify({prompt: hugeText, model_size:'4B', max_tokens:4096})})\n// after\nfetch('/llm/generate', {method:'POST', body: JSON.stringify({prompt: hugeText.slice(0,4000), model_size:'0.6B', max_tokens:512})})","handlingStrategy":"try-catch","validationCode":"// Pre-flight: confirm the LLM is loaded and a smaller size is available\nconst status = await (await fetch('/models/status')).json();\nconst llm = status.models.find(m => m.engine === 'qwen_llm' && m.loaded);\nif (!llm) {\n  // load via /models/download, then poll /models/progress until complete\n  throw new Error('LLM not loaded — start download first');\n}","typeGuard":null,"tryCatchPattern":"let lastErr;\nfor (const size of ['0.6B','1.7B','4B']) {\n  try {\n    const r = await fetch('/llm/generate', {method:'POST', body: JSON.stringify({...body, model_size: size})});\n    if (r.ok) return await r.json();\n    if (r.status !== 500) { lastErr = await r.json(); break; }\n    lastErr = await r.json().catch(() => ({}));\n  } catch (e) { lastErr = e; }\n}\nthrow new Error('LLM generation failed: ' + (lastErr?.detail ?? 'unknown'));","preventionTips":["Always surface the sanitized 500 as 'generation failed — see server logs', never retry blindly in a tight loop.","Trim prompt + examples before sending to reduce the chance of context-window/OOM faults.","Avoid issuing /llm/generate concurrently with /models/{name}/unload on the same model."],"tags":["llm","inference","http-500","oom","sanitized-error"],"backgroundTag":null,"analyzedSha":"51f49dea198384b4eb6087b72c17057c6eb1c1cd","analyzedAt":"2026-08-12T16:51:42.824Z","schemaVersion":2},"datasetVersion":"2026-08-12T18:17:37.767Z"}