{"record":{"id":"bc3eff1c2c1d0ede","repo":"mlflow/mlflow","slug":"failed-to-parse-json-response-was-response","errorCode":null,"errorMessage":"Failed to parse JSON. Response was: {response}","messagePattern":"Failed to parse JSON\\. Response was: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"mlflow/genai/scorers/ragas/models.py","lineNumber":81,"sourceCode":"\ndef _build_json_prompt(prompt: str, response_model: type[T]) -> str:\n    schema = response_model.model_json_schema()\n    fields = schema.get(\"properties\", {})\n    field_desc = \", \".join(f'\"{k}\"' for k in fields.keys())\n    return (\n        f\"{prompt}\\n\\n\"\n        f\"OUTPUT FORMAT: Respond ONLY with a JSON object \"\n        f\"containing these fields: {field_desc}, no other text. \"\n        f\"Do not add markdown formatting to the response.\"\n    )\n\n\ndef _parse_json_response(response: str, response_model: type[T]) -> T:\n    text = _strip_markdown_code_blocks(response)\n    try:\n        return response_model.model_validate(json.loads(text))\n    except json.JSONDecodeError as e:\n        raise ValueError(f\"Failed to parse JSON. Response was: {response}\") from e\n","sourceCodeStart":63,"sourceCodeEnd":82,"githubUrl":"https://github.com/mlflow/mlflow/blob/6a27f2decc0b76eb1b54af31849784addb357dbc/mlflow/genai/scorers/ragas/models.py#L63-L82","documentation":"When using a RAGAS adapter backed by a custom/MLflow LLM, the model's text completion is expected to be JSON matching a pydantic response model. _parse_json_response strips markdown code fences, then json.loads + model_validate; a non-JSON or malformed reply raises this ValueError including the raw response.","triggerScenarios":"generate() calls the LLM, gets a free-text reply (prose, apology, truncated output, or text plus JSON), and after stripping code blocks json.loads still fails — JSONDecodeError is re-raised as this message.","commonSituations":"Using a weak/small judge model that doesn't reliably emit JSON; the LLM refuses the task or returns chatty text; temperature too high; response truncated by max_tokens so JSON is cut off.","solutions":["Switch to a stronger, instruction-following judge model that reliably emits JSON (e.g. gpt-4o-class models).","Lower temperature / raise max_tokens for the judge so the JSON is complete and deterministic.","Prompt-constrain the output (the RAGAS prompt already asks for JSON; ensure no custom system prompt overrides it) and retry transient failures.","Pre-validate the raw response with a json.loads guard so you can retry on parse failure instead of crashing."],"exampleFix":"// before\ncustom_judge(model='ollama/tinyllama')  # prose output -> ValueError\n// after\ncustom_judge(model='openai:gpt-4o', temperature=0)","handlingStrategy":"retry","validationCode":"import json\ndef response_looks_like_json(response: str) -> bool:\n    text = response.strip().removeprefix('```json').removeprefix('```').removesuffix('```')\n    try:\n        json.loads(text)\n        return True\n    except json.JSONDecodeError:\n        return False","typeGuard":null,"tryCatchPattern":"import json\nfrom mlflow.exceptions import MlflowException\nfor attempt in range(3):\n    try:\n        return ragas_scorer(sample)\n    except ValueError as e:\n        if 'Failed to parse JSON' in str(e) and attempt < 2:\n            continue  # regenerate with a fresh LLM call\n        raise","preventionTips":["Use a strong instruction-following judge model for RAGAS metrics","Set temperature=0 and sufficient max_tokens for the judge","Keep default RAGAS prompts that instruct JSON output","Retry on JSON parse failures before failing the run"],"tags":["genai","ragas","json","llm-output-parsing"],"backgroundTag":"invalid-json-response","analyzedSha":"6a27f2decc0b76eb1b54af31849784addb357dbc","analyzedAt":"2026-08-29T20:54:51.419Z","schemaVersion":2},"datasetVersion":"2026-08-29T22:17:34.462Z"}