sherlock-project/sherlock · critical · FileNotFoundError

Bad response while accessing data file URL '{data_file_path}

Error message

Bad response while accessing data file URL '{data_file_path}'.

What it means

After successfully connecting to the manifest URL, SitesInformation checks response.status_code and requires exactly 200; any other status (301 that requests did not follow, 403, 404, 429, 5xx) raises FileNotFoundError with this message. It signals 'the server answered, but not with the manifest'.

Source

Thrown at sherlock_project/sites.py:134

        """

        if not data_file_path:
            # The default data file is the live data.json which is in the GitHub repo. The reason why we are using
            # this instead of the local one is so that the user has the most up-to-date data. This prevents
            # users from creating issue about false positives which has already been fixed or having outdated data
            data_file_path = MANIFEST_URL

        if data_file_path.lower().startswith("http"):
            # Reference is to a URL.
            try:
                response = requests.get(url=data_file_path, timeout=30)
            except Exception as error:
                raise FileNotFoundError(
                    f"Problem while attempting to access data file URL '{data_file_path}':  {error}"
                )

            if response.status_code != 200:
                raise FileNotFoundError(f"Bad response while accessing "
                                        f"data file URL '{data_file_path}'."
                                        )
            try:
                site_data = response.json()
            except Exception as error:
                raise ValueError(
                    f"Problem parsing json contents at '{data_file_path}':  {error}."
                )

        else:
            # Reference is to a file.
            try:
                with open(data_file_path, "r", encoding="utf-8") as file:
                    try:
                        site_data = json.load(file)
                    except Exception as error:
                        raise ValueError(
                            f"Problem parsing json contents at '{data_file_path}':  {error}."

View on GitHub (pinned to 9100f9d40a)

Solutions

  1. Reproduce with curl to see the actual status: `curl -o /dev/null -w '%{http_code}\n' <manifest-url>`; a 404 means a wrong URL/path, 429 means back off and retry later.
  2. For GitHub-hosted manifests use the raw content URL (raw.githubusercontent.com/<org>/<repo>/<ref>/<path>.json), not the HTML page.
  3. Retry with backoff on 429/5xx, or pin a specific commit ref of the manifest so the URL stays stable.
  4. Fall back to a local manifest copy: `sherlock --site ./data.json username` to keep working during an upstream outage.

Example fix

# before
sites = SitesInformation(data_file_path="https://github.com/me/sherlock/blob/main/data.json")
# -> HTML page URL, server answers but manifest path is not raw content

# after
sites = SitesInformation(data_file_path="https://raw.githubusercontent.com/me/sherlock/main/data.json")
Defensive patterns

Strategy: retry

Validate before calling

import requests

def manifest_responds_ok(url: str) -> bool:
    """Pre-flight: manifest must answer 200 before SitesInformation is built."""
    try:
        return requests.get(url, timeout=30).status_code == 200
    except requests.RequestException:
        return False

Try / catch

import time, requests
from sherlock_project.sites import SitesInformation

def load_sites(url, attempts=3, backoff=5):
    for i in range(attempts):
        try:
            return SitesInformation(data_file_path=url)
        except FileNotFoundError as err:
            # 429/5xx from the manifest host are transient: retry with backoff
            if i == attempts - 1:
                raise
            time.sleep(backoff * (2 ** i))

Prevention

When it happens

Trigger: The manifest host returns 403/404 because the URL or path in a custom data_file_path is wrong; 429 rate-limiting after repeated sherlock invocations; 5xx during an outage of data.sherlockproject.xyz; a captive portal returning 302/200-with-HTML (that case trips the JSON parse error instead); a custom --site URL served with a redirect loop or auth requirement (401).

Common situations: Pointing --site at a GitHub 'view' URL (github.com/...) instead of a raw.githubusercontent.com URL; API gateways returning 403 to non-browser user agents; rate limits in CI loops that hit the manifest on every run; the upstream data host being redeployed/renamed.

Related errors

AI-assisted analysis of sherlock-project/sherlock@9100f9d40a (2026-08-14). Data as JSON: /api/errors/59fa5480c6f498ab. Report an issue: GitHub.