{"record":{"id":"c052a60904506e27","repo":"odysseus-dev/odysseus","slug":"ai-returned-invalid-response","errorCode":null,"errorMessage":"AI returned invalid response","messagePattern":"AI returned invalid response","errorType":"http","errorClass":"HTTPException","httpStatus":500,"severity":"error","filePath":"routes/document/document_routes.py","lineNumber":1028,"sourceCode":"                \"No explanation, no markdown, just the JSON array.\\n\\n\"\n                + \"\\n\".join(doc_list)\n            )\n\n            response = await llm_call_async(\n                url, model,\n                [{\"role\": \"system\", \"content\": \"You classify documents as junk or keep. Respond only with a JSON array.\"},\n                 {\"role\": \"user\", \"content\": prompt}],\n                temperature=0.1,\n                max_tokens=200,\n                headers=headers,\n                timeout=30,\n            )\n\n            # Parse verdicts\n            import re\n            match = re.search(r'\\[.*?\\]', response, re.DOTALL)\n            if not match:\n                raise HTTPException(500, \"AI returned invalid response\")\n\n            import json as _json\n            verdicts = _json.loads(match.group())\n\n            deleted = 0\n            reviewed = 0\n            for i, doc in enumerate(batch):\n                if i >= len(verdicts):\n                    break\n                verdict = str(verdicts[i] or \"\").lower().strip()\n                if verdict == \"junk\":\n                    doc.tidy_verdict = \"junk\"\n                    db.delete(doc)\n                    deleted += 1\n                else:\n                    doc.tidy_verdict = \"keep\"\n                reviewed += 1\n","sourceCodeStart":1010,"sourceCodeEnd":1046,"githubUrl":"https://github.com/odysseus-dev/odysseus/blob/f9235ebbf13f693a6fd29ce70b097f6ec83705bf/routes/document/document_routes.py#L1010-L1046","documentation":"Raised inside POST /api/documents/ai-tidy when the LLM response for a document-classification batch does not contain any '[...]' JSON-array pattern (regex \\[.*?\\] with re.DOTALL). The model was asked to respond only with a JSON array of junk/keep verdicts; anything else — prose, an empty reply, a truncated array exceeding max_tokens=200, or an error string returned in place of content — triggers this 500.","triggerScenarios":"A batch of many documents makes the verdict array exceed the 200-token cap and get truncated to no closing bracket; the model wraps output in markdown or commentary so no bracketed array is parseable; the endpoint returns an error message string as the response body.","commonSituations":"Small max_tokens combined with large batches (verdicts are cheap but the array plus any preamble is not); switching LLM providers to one that ignores 'respond only with' instructions; gateway error bodies surfacing as response text.","solutions":["Reduce the batch size so the verdict array fits comfortably within max_tokens, or raise max_tokens.","Retry — temperature=0.1 sampling occasionally derails format; a retry often parses.","Check that the endpoint/model actually returns completion text (not an error payload) by logging the raw response once.","Tighten the prompt or switch to a model with reliable JSON-mode output."],"exampleFix":"# before\nresp = llm_call_async(messages, temperature=0.1, max_tokens=200, ...)\nmatch = re.search(r'\\[.*?\\]', response, re.DOTALL)\nif not match: raise HTTPException(500, \"AI returned invalid response\")\n\n# after\nresp = llm_call_async(messages, temperature=0.1, max_tokens=200*4, ...)\nmatch = re.search(r'\\[.*\\]', response, re.DOTALL)  # greedy to catch truncated arrays\nif not match:  # one bounded retry before failing\n    resp = llm_call_async(messages, temperature=0.0, max_tokens=200*4, ...)\n    match = re.search(r'\\[.*\\]', resp, re.DOTALL)\nif not match: raise HTTPException(500, \"AI returned invalid response\")","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"for attempt in range(2):\n    r = requests.post(f\"{base}/api/documents/ai-tidy\", timeout=120)\n    if r.status_code != 500 or \"invalid response\" not in r.json().get(\"detail\", \"\"):\n        break\n    time.sleep(2 ** attempt)  # re-prompt with smaller batch server-side","preventionTips":["Keep verdict batches small enough for the max_tokens cap.","Use JSON-mode-capable models for classification calls.","Treat format failures as transient — one retry usually parses."],"tags":["fastapi","llm","parsing","http-500","retry"],"backgroundTag":null,"analyzedSha":"f9235ebbf13f693a6fd29ce70b097f6ec83705bf","analyzedAt":"2026-08-14T21:47:48.359Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}