What causes "Failed to parse JSON. Response was: {response}" in mlflow/mlflow?

Switch to a stronger, instruction-following judge model that reliably emits JSON (e.g. gpt-4o-class models).

mlflow/mlflow · error · ValueError

Failed to parse JSON. Response was: {response}

Error message

Failed to parse JSON. Response was: {response}

What it means

When using a RAGAS adapter backed by a custom/MLflow LLM, the model's text completion is expected to be JSON matching a pydantic response model. _parse_json_response strips markdown code fences, then json.loads + model_validate; a non-JSON or malformed reply raises this ValueError including the raw response.

Source

Thrown at mlflow/genai/scorers/ragas/models.py:81

def _build_json_prompt(prompt: str, response_model: type[T]) -> str:
    schema = response_model.model_json_schema()
    fields = schema.get("properties", {})
    field_desc = ", ".join(f'"{k}"' for k in fields.keys())
    return (
        f"{prompt}\n\n"
        f"OUTPUT FORMAT: Respond ONLY with a JSON object "
        f"containing these fields: {field_desc}, no other text. "
        f"Do not add markdown formatting to the response."
    )


def _parse_json_response(response: str, response_model: type[T]) -> T:
    text = _strip_markdown_code_blocks(response)
    try:
        return response_model.model_validate(json.loads(text))
    except json.JSONDecodeError as e:
        raise ValueError(f"Failed to parse JSON. Response was: {response}") from e

View on GitHub (pinned to 6a27f2decc)

Solutions

Switch to a stronger, instruction-following judge model that reliably emits JSON (e.g. gpt-4o-class models).
Lower temperature / raise max_tokens for the judge so the JSON is complete and deterministic.
Prompt-constrain the output (the RAGAS prompt already asks for JSON; ensure no custom system prompt overrides it) and retry transient failures.
Pre-validate the raw response with a json.loads guard so you can retry on parse failure instead of crashing.

Example fix

// before
custom_judge(model='ollama/tinyllama')  # prose output -> ValueError
// after
custom_judge(model='openai:gpt-4o', temperature=0)

Defensive patterns

Strategy: retry

Validate before calling

import json
def response_looks_like_json(response: str) -> bool:
    text = response.strip().removeprefix('```json').removeprefix('```').removesuffix('```')
    try:
        json.loads(text)
        return True
    except json.JSONDecodeError:
        return False

Try / catch

import json
from mlflow.exceptions import MlflowException
for attempt in range(3):
    try:
        return ragas_scorer(sample)
    except ValueError as e:
        if 'Failed to parse JSON' in str(e) and attempt < 2:
            continue  # regenerate with a fresh LLM call
        raise

Prevention

Use a strong instruction-following judge model for RAGAS metrics
Set temperature=0 and sufficient max_tokens for the judge
Keep default RAGAS prompts that instruct JSON output
Retry on JSON parse failures before failing the run

When it happens

Trigger: generate() calls the LLM, gets a free-text reply (prose, apology, truncated output, or text plus JSON), and after stripping code blocks json.loads still fails — JSONDecodeError is re-raised as this message.

Common situations: Using a weak/small judge model that doesn't reliably emit JSON; the LLM refuses the task or returns chatty text; temperature too high; response truncated by max_tokens so JSON is cut off.

Understand the failure class

Background: "Invalid JSON response" and "Failed to parse response" errors: when an API answers 200 but the body isn't the JSON your library expected — this error's family across 28 libraries.

Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

AI-assisted analysis of mlflow/mlflow@6a27f2decc (2026-08-29). Data as JSON: /api/errors/bc3eff1c2c1d0ede. Report an issue: GitHub.