{"record":{"id":"3c243e5c4e6758c2","repo":"sherlock-project/sherlock","slug":"problem-parsing-json-contents-at-data-file-path","errorCode":null,"errorMessage":"Problem parsing json contents at '{data_file_path}':  {error}.","messagePattern":"Problem parsing json contents at '(.+?)':  (.+?)\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sherlock_project/sites.py","lineNumber":140,"sourceCode":"            data_file_path = MANIFEST_URL\n\n        if data_file_path.lower().startswith(\"http\"):\n            # Reference is to a URL.\n            try:\n                response = requests.get(url=data_file_path, timeout=30)\n            except Exception as error:\n                raise FileNotFoundError(\n                    f\"Problem while attempting to access data file URL '{data_file_path}':  {error}\"\n                )\n\n            if response.status_code != 200:\n                raise FileNotFoundError(f\"Bad response while accessing \"\n                                        f\"data file URL '{data_file_path}'.\"\n                                        )\n            try:\n                site_data = response.json()\n            except Exception as error:\n                raise ValueError(\n                    f\"Problem parsing json contents at '{data_file_path}':  {error}.\"\n                )\n\n        else:\n            # Reference is to a file.\n            try:\n                with open(data_file_path, \"r\", encoding=\"utf-8\") as file:\n                    try:\n                        site_data = json.load(file)\n                    except Exception as error:\n                        raise ValueError(\n                            f\"Problem parsing json contents at '{data_file_path}':  {error}.\"\n                        )\n\n            except FileNotFoundError:\n                raise FileNotFoundError(f\"Problem while attempting to access \"\n                                        f\"data file '{data_file_path}'.\"\n                                        )","sourceCodeStart":122,"sourceCodeEnd":158,"githubUrl":"https://github.com/sherlock-project/sherlock/blob/9100f9d40a3274bd46f4ce903c5c6fee6f3745bc/sherlock_project/sites.py#L122-L158","documentation":"The manifest URL returned HTTP 200, but response.json() failed to decode the body as JSON, so sites.py wraps the decode error (json.JSONDecodeError) in a ValueError. It means the bytes received were not a valid JSON document — typically an HTML error page, a login/captive-portal page, or a truncated body.","triggerScenarios":"A captive portal or 'accept cookies' interstitial answering 200 with HTML; a CDN serving a block page (Cloudflare-style challenge) with status 200; a custom data_file_path URL that returns an HTML index page instead of the raw file (classic GitHub blob-vs-raw mistake); a man-in-the-middle proxy rewriting responses; a manifest actually formatted as JSON5/YAML or containing a BOM/comments.","commonSituations":"Public Wi-Fi captive portals; corporate TLS-termination proxies injecting notices; pointing --site at a web UI page rather than the raw file; hand-authored manifests saved with a UTF-8 BOM or trailing commas; a partially-written manifest uploaded mid-deploy.","solutions":["Fetch the URL manually (`curl -s <url> | python -m json.tool`) — the exact decode error and the first offending bytes will show whether you got HTML instead of JSON.","If HTML came back, switch to the raw content URL (e.g. raw.githubusercontent.com for GitHub) or fix the server to serve the file with correct content.","Remove JSON-invalid syntax from a hand-edited manifest: BOMs, comments, trailing commas, single quotes; validate with `python -m json.tool data.json`.","For captive portals / proxies, complete authentication or bypass the network, then retry."],"exampleFix":"# before\nsites = SitesInformation(data_file_path=\"https://example.com/my-list\")  # serves HTML\n\n# after\nimport json, requests\nr = requests.get(\"https://example.com/my-list.json\", timeout=30)\njson.loads(r.text)  # fail fast with the real JSON error before constructing SitesInformation\nsites = SitesInformation(data_file_path=\"https://example.com/my-list.json\")","handlingStrategy":"validation","validationCode":"import json, requests\n\ndef fetch_manifest_json(url: str) -> dict:\n    \"\"\"Fetch and JSON-validate the manifest before handing it to sherlock.\"\"\"\n    resp = requests.get(url, timeout=30)\n    resp.raise_for_status()\n    try:\n        return resp.json()\n    except json.JSONDecodeError as err:\n        snippet = resp.text[:120].replace(\"\\n\", \" \")\n        raise ValueError(f\"Manifest at {url} is not JSON (starts with: {snippet!r})\") from err","typeGuard":null,"tryCatchPattern":"from sherlock_project.sites import SitesInformation\n\ntry:\n    sites = SitesInformation(data_file_path=url)\nexcept ValueError as err:\n    # Covers JSON decode failure of a URL manifest; err names the path and cause.\n    # Response was 200 but the body was not JSON (portal/block page/HTML) —\n    # inspect the URL in a browser or curl, then retry or switch to a local file.\n    raise","preventionTips":["Pre-flight the manifest with curl | python -m json.tool in setup scripts to catch HTML-instead-of-JSON early.","On public Wi-Fi, confirm internet access past captive portals before running sherlock.","Serve manifests with an explicit application/json content type and no BOM from your own hosts.","Keep a validated local snapshot for environments where middleboxes rewrite responses."],"tags":["sherlock","json","manifest","captive-portal","parsing"],"backgroundTag":null,"analyzedSha":"9100f9d40a3274bd46f4ce903c5c6fee6f3745bc","analyzedAt":"2026-08-14T19:48:26.535Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}