{"record":{"id":"ae204375a653ac14","repo":"datawhalechina/hello-agents","slug":"pipeline-error-e","errorCode":null,"errorMessage":"Pipeline error: {e}","messagePattern":"Pipeline error: (.+?)","errorType":"http","errorClass":"HTTPException","httpStatus":500,"severity":"error","filePath":"Co-creation-projects/zjzhou-SREOnCallAgent/src/api/main.py","lineNumber":69,"sourceCode":"\n\n@app.post(\"/incidents/investigate\")\ndef investigate(req: InvestigateRequest):\n    \"\"\"\n    Run the full triage → investigation → post-mortem pipeline for an incident.\n\n    This runs synchronously (suitable for demo; upgrade to background task + SSE for prod).\n    \"\"\"\n    try:\n        load_incident(req.incident_id)  # Validate early\n    except FileNotFoundError as e:\n        raise HTTPException(status_code=404, detail=str(e))\n\n    start = time.time()\n    try:\n        result = run_pipeline(req.incident_id, verbose=False)\n    except Exception as e:\n        raise HTTPException(status_code=500, detail=f\"Pipeline error: {e}\")\n\n    elapsed = round(time.time() - start, 1)\n    result[\"elapsed_seconds\"] = elapsed\n    _report_store[req.incident_id] = result\n    return result\n\n\n@app.get(\"/incidents/{incident_id}/report\")\ndef get_report(incident_id: str):\n    \"\"\"Retrieve a previously generated post-mortem report.\"\"\"\n    if incident_id not in _report_store:\n        raise HTTPException(\n            status_code=404,\n            detail=f\"No report found for '{incident_id}'. Call POST /incidents/investigate first.\",\n        )\n    return {\n        \"incident_id\": incident_id,\n        \"report\": _report_store[incident_id][\"report\"],","sourceCodeStart":51,"sourceCodeEnd":87,"githubUrl":"https://github.com/datawhalechina/hello-agents/blob/606a07d341a47be773fab7f4b71177f53f96b2c3/Co-creation-projects/zjzhou-SREOnCallAgent/src/api/main.py#L51-L87","documentation":"The /incidents/investigate endpoint wraps any exception from run_pipeline (triage → investigation → post-mortem) into HTTPException(500, f\"Pipeline error: {e}\"). The LLM stages call out to models, so the embedded message is usually an underlying provider error — auth, rate limit, timeout, or malformed response — rather than a bug in the pipeline code itself.","triggerScenarios":"POST /incidents/investigate with a valid incident ID while the LLM API key is invalid/expired, the model endpoint is unreachable, a rate limit trips mid-run, or an agent returns unparseable output. Because the run is synchronous, long pipelines can also die on read timeouts.","commonSituations":"Expired provider API key in the deployment env; egress blocked in containerized environments; 429s when several investigations run concurrently; model/schema drift after a provider deprecates a response format.","solutions":["Read the detail text after 'Pipeline error:' — fix that root cause (auth, network, rate limit) first.","Retry with backoff for transient causes (429/timeout); the endpoint is idempotent per incident.","Verify provider credentials and connectivity from the API host (curl the model endpoint).","If timeouts dominate, raise the HTTP client timeout or move to the suggested background-task + SSE design noted in the endpoint docstring."],"exampleFix":"# before\nresp = requests.post(url, json={\"incident_id\": \"inc-001\"})\nassert resp.status_code == 200\n\n# after\nfor attempt in range(3):\n    resp = requests.post(url, json={\"incident_id\": \"inc-001\"})\n    if resp.status_code != 500 or attempt == 2:\n        break\n    time.sleep(2 ** attempt)\nprint(resp.json())","handlingStrategy":"retry","validationCode":"# preflight provider connectivity from the API host before a long pipeline\nimport os, requests\n\nrequests.post(\n    os.environ[\"LLM_BASE_URL\"].rstrip(\"/\") + \"/chat/completions\",\n    headers={\"Authorization\": f\"Bearer {os.environ['LLM_API_KEY']}\"},\n    json={\"model\": os.environ[\"LLM_MODEL_ID\"], \"messages\": [{\"role\": \"user\", \"content\": \"ping\"}]},\n    timeout=15,\n).raise_for_status()","typeGuard":null,"tryCatchPattern":"import time, requests\n\nfor attempt in range(3):\n    resp = requests.post(url, json={\"incident_id\": iid}, timeout=300)\n    if resp.status_code != 500 or attempt == 2:\n        break\n    detail = resp.json().get(\"detail\", \"\")\n    if not any(t in detail for t in (\"429\", \"timeout\", \"timed out\", \"Connection\")):\n        break  # non-transient: escalate immediately\n    time.sleep(2 ** attempt)\nresp.raise_for_status()","preventionTips":["Distinguish transient (429/timeout/network) from permanent (401/404 model) 500-detail messages before retrying.","Set client timeouts longer than the slowest LLM stage; the endpoint is synchronous.","Move to the background-task + SSE design noted in the endpoint docstring for production.","Alert on 500-rate spikes — provider outages surface here first.","Keep pipeline idempotent per incident so retries are safe."],"tags":["fastapi","http-500","llm","sre"],"backgroundTag":null,"analyzedSha":"606a07d341a47be773fab7f4b71177f53f96b2c3","analyzedAt":"2026-08-14T22:57:27.446Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}