odysseus-dev/odysseus · error · HTTPException

AI returned invalid response

Error message

AI returned invalid response

What it means

Raised inside POST /api/documents/ai-tidy when the LLM response for a document-classification batch does not contain any '[...]' JSON-array pattern (regex \[.*?\] with re.DOTALL). The model was asked to respond only with a JSON array of junk/keep verdicts; anything else — prose, an empty reply, a truncated array exceeding max_tokens=200, or an error string returned in place of content — triggers this 500.

Source

Thrown at routes/document/document_routes.py:1028

                "No explanation, no markdown, just the JSON array.\n\n"
                + "\n".join(doc_list)
            )

            response = await llm_call_async(
                url, model,
                [{"role": "system", "content": "You classify documents as junk or keep. Respond only with a JSON array."},
                 {"role": "user", "content": prompt}],
                temperature=0.1,
                max_tokens=200,
                headers=headers,
                timeout=30,
            )

            # Parse verdicts
            import re
            match = re.search(r'\[.*?\]', response, re.DOTALL)
            if not match:
                raise HTTPException(500, "AI returned invalid response")

            import json as _json
            verdicts = _json.loads(match.group())

            deleted = 0
            reviewed = 0
            for i, doc in enumerate(batch):
                if i >= len(verdicts):
                    break
                verdict = str(verdicts[i] or "").lower().strip()
                if verdict == "junk":
                    doc.tidy_verdict = "junk"
                    db.delete(doc)
                    deleted += 1
                else:
                    doc.tidy_verdict = "keep"
                reviewed += 1

View on GitHub (pinned to f9235ebbf1)

Solutions

  1. Reduce the batch size so the verdict array fits comfortably within max_tokens, or raise max_tokens.
  2. Retry — temperature=0.1 sampling occasionally derails format; a retry often parses.
  3. Check that the endpoint/model actually returns completion text (not an error payload) by logging the raw response once.
  4. Tighten the prompt or switch to a model with reliable JSON-mode output.

Example fix

# before
resp = llm_call_async(messages, temperature=0.1, max_tokens=200, ...)
match = re.search(r'\[.*?\]', response, re.DOTALL)
if not match: raise HTTPException(500, "AI returned invalid response")

# after
resp = llm_call_async(messages, temperature=0.1, max_tokens=200*4, ...)
match = re.search(r'\[.*\]', response, re.DOTALL)  # greedy to catch truncated arrays
if not match:  # one bounded retry before failing
    resp = llm_call_async(messages, temperature=0.0, max_tokens=200*4, ...)
    match = re.search(r'\[.*\]', resp, re.DOTALL)
if not match: raise HTTPException(500, "AI returned invalid response")
Defensive patterns

Strategy: retry

Try / catch

for attempt in range(2):
    r = requests.post(f"{base}/api/documents/ai-tidy", timeout=120)
    if r.status_code != 500 or "invalid response" not in r.json().get("detail", ""):
        break
    time.sleep(2 ** attempt)  # re-prompt with smaller batch server-side

Prevention

When it happens

Trigger: A batch of many documents makes the verdict array exceed the 200-token cap and get truncated to no closing bracket; the model wraps output in markdown or commentary so no bracketed array is parseable; the endpoint returns an error message string as the response body.

Common situations: Small max_tokens combined with large batches (verdicts are cheap but the array plus any preamble is not); switching LLM providers to one that ignores 'respond only with' instructions; gateway error bodies surfacing as response text.

Related errors

AI-assisted analysis of odysseus-dev/odysseus@f9235ebbf1 (2026-08-14). Data as JSON: /api/errors/c052a60904506e27. Report an issue: GitHub.