What causes "Pipeline error: {e}" in datawhalechina/hello-agents?

Read the detail text after 'Pipeline error:' — fix that root cause (auth, network, rate limit) first.

datawhalechina/hello-agents · error · HTTPException

Pipeline error: {e}

Error message

Pipeline error: {e}

What it means

The /incidents/investigate endpoint wraps any exception from run_pipeline (triage → investigation → post-mortem) into HTTPException(500, f"Pipeline error: {e}"). The LLM stages call out to models, so the embedded message is usually an underlying provider error — auth, rate limit, timeout, or malformed response — rather than a bug in the pipeline code itself.

Source

Thrown at Co-creation-projects/zjzhou-SREOnCallAgent/src/api/main.py:69


@app.post("/incidents/investigate")
def investigate(req: InvestigateRequest):
    """
    Run the full triage → investigation → post-mortem pipeline for an incident.

    This runs synchronously (suitable for demo; upgrade to background task + SSE for prod).
    """
    try:
        load_incident(req.incident_id)  # Validate early
    except FileNotFoundError as e:
        raise HTTPException(status_code=404, detail=str(e))

    start = time.time()
    try:
        result = run_pipeline(req.incident_id, verbose=False)
    except Exception as e:
        raise HTTPException(status_code=500, detail=f"Pipeline error: {e}")

    elapsed = round(time.time() - start, 1)
    result["elapsed_seconds"] = elapsed
    _report_store[req.incident_id] = result
    return result


@app.get("/incidents/{incident_id}/report")
def get_report(incident_id: str):
    """Retrieve a previously generated post-mortem report."""
    if incident_id not in _report_store:
        raise HTTPException(
            status_code=404,
            detail=f"No report found for '{incident_id}'. Call POST /incidents/investigate first.",
        )
    return {
        "incident_id": incident_id,
        "report": _report_store[incident_id]["report"],

View on GitHub (pinned to 606a07d341)

Solutions

Read the detail text after 'Pipeline error:' — fix that root cause (auth, network, rate limit) first.
Retry with backoff for transient causes (429/timeout); the endpoint is idempotent per incident.
Verify provider credentials and connectivity from the API host (curl the model endpoint).
If timeouts dominate, raise the HTTP client timeout or move to the suggested background-task + SSE design noted in the endpoint docstring.

Example fix

# before
resp = requests.post(url, json={"incident_id": "inc-001"})
assert resp.status_code == 200

# after
for attempt in range(3):
    resp = requests.post(url, json={"incident_id": "inc-001"})
    if resp.status_code != 500 or attempt == 2:
        break
    time.sleep(2 ** attempt)
print(resp.json())

Defensive patterns

Strategy: retry

Validate before calling

# preflight provider connectivity from the API host before a long pipeline
import os, requests

requests.post(
    os.environ["LLM_BASE_URL"].rstrip("/") + "/chat/completions",
    headers={"Authorization": f"Bearer {os.environ['LLM_API_KEY']}"},
    json={"model": os.environ["LLM_MODEL_ID"], "messages": [{"role": "user", "content": "ping"}]},
    timeout=15,
).raise_for_status()

Try / catch

import time, requests

for attempt in range(3):
    resp = requests.post(url, json={"incident_id": iid}, timeout=300)
    if resp.status_code != 500 or attempt == 2:
        break
    detail = resp.json().get("detail", "")
    if not any(t in detail for t in ("429", "timeout", "timed out", "Connection")):
        break  # non-transient: escalate immediately
    time.sleep(2 ** attempt)
resp.raise_for_status()

Prevention

Distinguish transient (429/timeout/network) from permanent (401/404 model) 500-detail messages before retrying.
Set client timeouts longer than the slowest LLM stage; the endpoint is synchronous.
Move to the background-task + SSE design noted in the endpoint docstring for production.
Alert on 500-rate spikes — provider outages surface here first.
Keep pipeline idempotent per incident so retries are safe.

When it happens

Trigger: POST /incidents/investigate with a valid incident ID while the LLM API key is invalid/expired, the model endpoint is unreachable, a rate limit trips mid-run, or an agent returns unparseable output. Because the run is synchronous, long pipelines can also die on read timeouts.

Common situations: Expired provider API key in the deployment env; egress blocked in containerized environments; 429s when several investigations run concurrently; model/schema drift after a provider deprecates a response format.

Related errors

AI-assisted analysis of datawhalechina/hello-agents@606a07d341 (2026-08-14). Data as JSON: /api/errors/ae204375a653ac14. Report an issue: GitHub.