assafelovic/gpt-researcher · error · Exception

Error parsing SearxNG response

Error message

Error parsing SearxNG response

What it means

Exception raised by SearxNG search() when the instance returns HTTP 200 but the body is not valid JSON (json.JSONDecodeError caught and re-raised as this plain message). Most commonly the instance returned an HTML page because JSON output is disabled for API clients.

Source

Thrown at gpt_researcher/retrievers/searx/searx.py:88

        params = {
            # The search query.
            'q': self.query,
            # Output format of results. Format needs to be activated in searxng config.
            'format': 'json'
        }

        try:
            response = requests.get(
                search_url,
                params=params,
                headers={'Accept': 'application/json'}
            )
            response.raise_for_status()
            results = response.json()
        except requests.exceptions.RequestException as e:
            raise Exception(f"Error querying SearxNG: {str(e)}")
        except json.JSONDecodeError:
            raise Exception("Error parsing SearxNG response")

        if not isinstance(results, dict):
            return []

        search_response = []
        raw_results = results.get('results', [])
        if not isinstance(raw_results, list):
            return []

        for result in raw_results:
            if not isinstance(result, dict):
                continue
            href = result.get('url') or result.get('href') or ''
            if not href:
                continue
            body = result.get('content') or result.get('snippet') or ''
            search_response.append({
                "href": href,

View on GitHub (pinned to 6f998577d5)

Solutions

1. Enable JSON output on your instance: in settings.yml set search: formats: [html, json] and restart.
2. If using a public instance, pick one from searx.space that allows format=json, or self-host.
3. Check response.text of the URL directly to see what HTML is being served (CAPTCHA/proxy error).
4. Ensure no proxy is rewriting the Accept: application/json request.

Example fix

# searx settings.yml — before
search:
  formats:
    - html

# after
search:
  formats:
    - html
    - json
# then restart searxng

import requests, os
url = os.environ["SEARX_URL"].rstrip('/') + '/search'
r = requests.get(url, params={'q': 'test', 'format': 'json'}, timeout=5,
                 headers={'Accept': 'application/json'})
assert r.headers.get('content-type', '').startswith('application/json'), \
    "SearxNG JSON output disabled — enable 'json' in search.formats"

null

try:
    results = retriever.search()
except Exception as e:
    if "Error parsing" in str(e):
        logger.error("SearxNG returned HTML; enable format=json on the instance")
        results = []
    else:
        raise

When it happens

Trigger: search() calls response.json(); the SearxNG instance responds with HTML (error page, CAPTCHA, or default HTML format) instead of the requested JSON.

Common situations: Self-hosted SearxNG without 'json' in search.formats, public instances that disable the JSON API to prevent scraping, or a reverse proxy/WAF serving an HTML error page.

Understand the failure class

Background: "Invalid JSON response" and "Failed to parse response" errors: when an API answers 200 but the body isn't the JSON your library expected — this error's family across 28 libraries.

Related errors

• MCPRetriever requires a researcher instance with cfg attribu
• SearxNG URL not found. Please set the SEARX_URL environment
• Error querying SearxNG: {str(e)}
• SerpApi API key not found. Please set the SERPAPI_API_KEY en
• Serper API key not found. Please set the SERPER_API_KEY envi

AI-assisted analysis of assafelovic/gpt-researcher@6f998577d5 (2026-08-28). Data as JSON: /api/errors/ec47b2bcf96e8eaf. Report an issue: GitHub.