{"record":{"id":"b241cabcb6515d35","repo":"firecrawl/firecrawl","slug":"failed-to-parse-firecrawl-response-as-json-b241ca","errorCode":null,"errorMessage":"Failed to parse Firecrawl response as JSON.","messagePattern":"Failed to parse Firecrawl response as JSON\\.","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"apps/python-sdk/firecrawl/v1/client.py","lineNumber":686,"sourceCode":"        # Make request\n        response = requests.post(\n            f'{self.api_url}/v1/scrape',\n            headers=_headers,\n            json=scrape_params,\n            timeout=(timeout / 1000.0 + 5 if timeout is not None else None)\n        )\n\n        if response.status_code == 200:\n            try:\n                response_json = response.json()\n                if response_json.get('success') and 'data' in response_json:\n                    return V1ScrapeResponse(**response_json['data'])\n                elif \"error\" in response_json:\n                    raise Exception(f'Failed to scrape URL. Error: {response_json[\"error\"]}')\n                else:\n                    raise Exception(f'Failed to scrape URL. Error: {response_json}')\n            except ValueError:\n                raise Exception('Failed to parse Firecrawl response as JSON.')\n        else:\n            self._handle_error(response, 'scrape URL')\n\n    def search(\n            self,\n            query: str,\n            *,\n            limit: Optional[int] = None,\n            tbs: Optional[str] = None,\n            filter: Optional[str] = None,\n            lang: Optional[str] = None,\n            country: Optional[str] = None,\n            location: Optional[str] = None,\n            timeout: Optional[int] = 30000,\n            scrape_options: Optional[V1ScrapeOptions] = None,\n            **kwargs) -> V1SearchResponse:\n        \"\"\"\n        Search for content using Firecrawl.","sourceCodeStart":668,"sourceCodeEnd":704,"githubUrl":"https://github.com/firecrawl/firecrawl/blob/656bffcc2883f1af5befe38766b1ff5f0469993a/apps/python-sdk/firecrawl/v1/client.py#L668-L704","documentation":"Raised by scrape_url when response.json() raises ValueError (Python's json decoder) on a 200 response. The body was not valid JSON, so the SDK cannot parse the scrape result.","triggerScenarios":"The server returns 200 with HTML/text (e.g. a gateway error page, a CAPTCHA page, or an empty body); a proxy injects non-JSON content; the connection was truncated mid-body.","commonSituations":"Cloudflare interstitial served with 200; misconfigured reverse proxy returning plain-text errors; intermittent network drops that truncate the response.","solutions":["Retry once - truncation and transient interstitials often clear on the next request.","Inspect response.text indirectly by wrapping scrape_url and dumping the raw body before parsing fails.","If using a proxy, verify it sets Accept: application/json and does not transform responses.","Report to firecrawl if the body is consistently non-JSON from api.firecrawl.dev."],"exampleFix":"// before\nresult = app.scrape_url(url)\n\n// after\nimport time\nfor attempt in range(3):\n    try:\n        result = app.scrape_url(url)\n        break\n    except Exception as e:\n        if \"Failed to parse\" in str(e) and attempt < 2:\n            time.sleep(1)\n            continue\n        raise","handlingStrategy":"retry","validationCode":"import requests\n# pre-flight: confirm endpoint returns JSON\nr = requests.head(f'{api_url}/v1/scrape', headers={'Authorization': f'Bearer {api_key}'})\nassert 'json' in r.headers.get('content-type', ''), 'endpoint not serving JSON'","typeGuard":"def looks_like_json(text: str) -> bool:\n    t = text.lstrip()[:1]\n    return t in ('{', '[')","tryCatchPattern":"import time\nfor attempt in range(3):\n    try:\n        result = app.scrape_url(url)\n        break\n    except Exception as e:\n        if 'Failed to parse' in str(e) and attempt < 2:\n            time.sleep(2 ** attempt)\n            continue\n        raise","preventionTips":["Implement a small retry-with-backoff around scrape_url to absorb transient non-JSON responses.","Use a dedicated HTTP client logger that dumps response.text on parse failures.","Avoid routing firecrawl traffic through transforming proxies."],"tags":["scrape","json-parse","network"],"analyzedSha":"656bffcc2883f1af5befe38766b1ff5f0469993a","analyzedAt":"2026-08-12T01:18:00.488Z","schemaVersion":2},"datasetVersion":"2026-08-12T06:17:24.410Z"}