{"record":{"id":"59fa5480c6f498ab","repo":"sherlock-project/sherlock","slug":"bad-response-while-accessing-data-file-url-data","errorCode":null,"errorMessage":"Bad response while accessing data file URL '{data_file_path}'.","messagePattern":"Bad response while accessing data file URL '(.+?)'\\.","errorType":"exception","errorClass":"FileNotFoundError","httpStatus":null,"severity":"critical","filePath":"sherlock_project/sites.py","lineNumber":134,"sourceCode":"        \"\"\"\n\n        if not data_file_path:\n            # The default data file is the live data.json which is in the GitHub repo. The reason why we are using\n            # this instead of the local one is so that the user has the most up-to-date data. This prevents\n            # users from creating issue about false positives which has already been fixed or having outdated data\n            data_file_path = MANIFEST_URL\n\n        if data_file_path.lower().startswith(\"http\"):\n            # Reference is to a URL.\n            try:\n                response = requests.get(url=data_file_path, timeout=30)\n            except Exception as error:\n                raise FileNotFoundError(\n                    f\"Problem while attempting to access data file URL '{data_file_path}':  {error}\"\n                )\n\n            if response.status_code != 200:\n                raise FileNotFoundError(f\"Bad response while accessing \"\n                                        f\"data file URL '{data_file_path}'.\"\n                                        )\n            try:\n                site_data = response.json()\n            except Exception as error:\n                raise ValueError(\n                    f\"Problem parsing json contents at '{data_file_path}':  {error}.\"\n                )\n\n        else:\n            # Reference is to a file.\n            try:\n                with open(data_file_path, \"r\", encoding=\"utf-8\") as file:\n                    try:\n                        site_data = json.load(file)\n                    except Exception as error:\n                        raise ValueError(\n                            f\"Problem parsing json contents at '{data_file_path}':  {error}.\"","sourceCodeStart":116,"sourceCodeEnd":152,"githubUrl":"https://github.com/sherlock-project/sherlock/blob/9100f9d40a3274bd46f4ce903c5c6fee6f3745bc/sherlock_project/sites.py#L116-L152","documentation":"After successfully connecting to the manifest URL, SitesInformation checks response.status_code and requires exactly 200; any other status (301 that requests did not follow, 403, 404, 429, 5xx) raises FileNotFoundError with this message. It signals 'the server answered, but not with the manifest'.","triggerScenarios":"The manifest host returns 403/404 because the URL or path in a custom data_file_path is wrong; 429 rate-limiting after repeated sherlock invocations; 5xx during an outage of data.sherlockproject.xyz; a captive portal returning 302/200-with-HTML (that case trips the JSON parse error instead); a custom --site URL served with a redirect loop or auth requirement (401).","commonSituations":"Pointing --site at a GitHub 'view' URL (github.com/...) instead of a raw.githubusercontent.com URL; API gateways returning 403 to non-browser user agents; rate limits in CI loops that hit the manifest on every run; the upstream data host being redeployed/renamed.","solutions":["Reproduce with curl to see the actual status: `curl -o /dev/null -w '%{http_code}\\n' <manifest-url>`; a 404 means a wrong URL/path, 429 means back off and retry later.","For GitHub-hosted manifests use the raw content URL (raw.githubusercontent.com/<org>/<repo>/<ref>/<path>.json), not the HTML page.","Retry with backoff on 429/5xx, or pin a specific commit ref of the manifest so the URL stays stable.","Fall back to a local manifest copy: `sherlock --site ./data.json username` to keep working during an upstream outage."],"exampleFix":"# before\nsites = SitesInformation(data_file_path=\"https://github.com/me/sherlock/blob/main/data.json\")\n# -> HTML page URL, server answers but manifest path is not raw content\n\n# after\nsites = SitesInformation(data_file_path=\"https://raw.githubusercontent.com/me/sherlock/main/data.json\")","handlingStrategy":"retry","validationCode":"import requests\n\ndef manifest_responds_ok(url: str) -> bool:\n    \"\"\"Pre-flight: manifest must answer 200 before SitesInformation is built.\"\"\"\n    try:\n        return requests.get(url, timeout=30).status_code == 200\n    except requests.RequestException:\n        return False","typeGuard":null,"tryCatchPattern":"import time, requests\nfrom sherlock_project.sites import SitesInformation\n\ndef load_sites(url, attempts=3, backoff=5):\n    for i in range(attempts):\n        try:\n            return SitesInformation(data_file_path=url)\n        except FileNotFoundError as err:\n            # 429/5xx from the manifest host are transient: retry with backoff\n            if i == attempts - 1:\n                raise\n            time.sleep(backoff * (2 ** i))","preventionTips":["Cache the last successful manifest and use it as the fallback after retries are exhausted.","Use raw content URLs (raw.githubusercontent.com/...) for GitHub-hosted manifests, never blob/view pages.","Add jittered backoff in CI loops so repeated runs do not trip 429 rate limits on the manifest host.","Pin a specific commit/tag of a third-party manifest so a moved or deleted path does not break your pipeline."],"tags":["sherlock","http-status","manifest","rate-limit","url"],"backgroundTag":null,"analyzedSha":"9100f9d40a3274bd46f4ce903c5c6fee6f3745bc","analyzedAt":"2026-08-14T19:48:26.535Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}