What causes "Failed to parse Firecrawl response as JSON." in firecrawl/firecrawl?

Retry once - truncation and transient interstitials often clear on the next request.

firecrawl/firecrawl · error · Exception

Failed to parse Firecrawl response as JSON.

Error message

Failed to parse Firecrawl response as JSON.

What it means

Raised by scrape_url when response.json() raises ValueError (Python's json decoder) on a 200 response. The body was not valid JSON, so the SDK cannot parse the scrape result.

Source

Thrown at apps/python-sdk/firecrawl/v1/client.py:686

        # Make request
        response = requests.post(
            f'{self.api_url}/v1/scrape',
            headers=_headers,
            json=scrape_params,
            timeout=(timeout / 1000.0 + 5 if timeout is not None else None)
        )

        if response.status_code == 200:
            try:
                response_json = response.json()
                if response_json.get('success') and 'data' in response_json:
                    return V1ScrapeResponse(**response_json['data'])
                elif "error" in response_json:
                    raise Exception(f'Failed to scrape URL. Error: {response_json["error"]}')
                else:
                    raise Exception(f'Failed to scrape URL. Error: {response_json}')
            except ValueError:
                raise Exception('Failed to parse Firecrawl response as JSON.')
        else:
            self._handle_error(response, 'scrape URL')

    def search(
            self,
            query: str,
            *,
            limit: Optional[int] = None,
            tbs: Optional[str] = None,
            filter: Optional[str] = None,
            lang: Optional[str] = None,
            country: Optional[str] = None,
            location: Optional[str] = None,
            timeout: Optional[int] = 30000,
            scrape_options: Optional[V1ScrapeOptions] = None,
            **kwargs) -> V1SearchResponse:
        """
        Search for content using Firecrawl.

View on GitHub (pinned to 656bffcc28)

Solutions

Retry once - truncation and transient interstitials often clear on the next request.
Inspect response.text indirectly by wrapping scrape_url and dumping the raw body before parsing fails.
If using a proxy, verify it sets Accept: application/json and does not transform responses.
Report to firecrawl if the body is consistently non-JSON from api.firecrawl.dev.

Example fix

// before
result = app.scrape_url(url)

// after
import time
for attempt in range(3):
    try:
        result = app.scrape_url(url)
        break
    except Exception as e:
        if "Failed to parse" in str(e) and attempt < 2:
            time.sleep(1)
            continue
        raise

Defensive patterns

Strategy: retry

Validate before calling

import requests
# pre-flight: confirm endpoint returns JSON
r = requests.head(f'{api_url}/v1/scrape', headers={'Authorization': f'Bearer {api_key}'})
assert 'json' in r.headers.get('content-type', ''), 'endpoint not serving JSON'

Type guard

def looks_like_json(text: str) -> bool:
    t = text.lstrip()[:1]
    return t in ('{', '[')

Try / catch

import time
for attempt in range(3):
    try:
        result = app.scrape_url(url)
        break
    except Exception as e:
        if 'Failed to parse' in str(e) and attempt < 2:
            time.sleep(2 ** attempt)
            continue
        raise

Prevention

Implement a small retry-with-backoff around scrape_url to absorb transient non-JSON responses.
Use a dedicated HTTP client logger that dumps response.text on parse failures.
Avoid routing firecrawl traffic through transforming proxies.

When it happens

Trigger: The server returns 200 with HTML/text (e.g. a gateway error page, a CAPTCHA page, or an empty body); a proxy injects non-JSON content; the connection was truncated mid-body.

Common situations: Cloudflare interstitial served with 200; misconfigured reverse proxy returning plain-text errors; intermittent network drops that truncate the response.

Understand the failure class

Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

AI-assisted analysis of firecrawl/firecrawl@656bffcc28 (2026-08-12). Data as JSON: /api/errors/b241cabcb6515d35. Report an issue: GitHub.