sherlock-project/sherlock · critical · FileNotFoundError

Problem while attempting to access data file URL '{data_file

Error message

Problem while attempting to access data file URL '{data_file_path}':  {error}

What it means

SitesInformation.__init__() loads the sites manifest, defaulting to the live MANIFEST_URL (https://data.sherlockproject.xyz) when data_file_path is empty. If requests.get() raises for any reason while fetching a manifest URL, the exception is caught, wrapped and re-raised as FileNotFoundError with this message. Despite the exception type, it almost always means a network-level failure, not a missing file.

Source

Thrown at sherlock_project/sites.py:129

                                  If this option is not specified, then a
                                  default site list will be used.

        Return Value:
        Nothing.
        """

        if not data_file_path:
            # The default data file is the live data.json which is in the GitHub repo. The reason why we are using
            # this instead of the local one is so that the user has the most up-to-date data. This prevents
            # users from creating issue about false positives which has already been fixed or having outdated data
            data_file_path = MANIFEST_URL

        if data_file_path.lower().startswith("http"):
            # Reference is to a URL.
            try:
                response = requests.get(url=data_file_path, timeout=30)
            except Exception as error:
                raise FileNotFoundError(
                    f"Problem while attempting to access data file URL '{data_file_path}':  {error}"
                )

            if response.status_code != 200:
                raise FileNotFoundError(f"Bad response while accessing "
                                        f"data file URL '{data_file_path}'."
                                        )
            try:
                site_data = response.json()
            except Exception as error:
                raise ValueError(
                    f"Problem parsing json contents at '{data_file_path}':  {error}."
                )

        else:
            # Reference is to a file.
            try:
                with open(data_file_path, "r", encoding="utf-8") as file:

View on GitHub (pinned to 9100f9d40a)

Solutions

Check connectivity to the manifest host first: `curl -I https://data.sherlockproject.xyz` and fix network/DNS/proxy issues (set HTTPS_PROXY if a proxy is required).
Pass a local manifest instead of the live one: `sherlock --site /path/to/data.json username` (any path not starting with http is read from disk), e.g. the copy shipped in the repo under sherlock_project/resources/.
If behind TLS interception, point REQUESTS_CA_BUNDLE at the corporate CA bundle.
If the failure is a timeout, retry — the 30s limit can trip on slow links; consider pinning a known-good manifest URL or local snapshot for reproducible runs.

Example fix

# before
sites = SitesInformation()  # fetches live manifest, fails offline

# after
import requests
try:
    sites = SitesInformation()
except FileNotFoundError:
    # fall back to the local copy shipped with the installed package
    from importlib import resources
    local = resources.files("sherlock_project") / "resources/data.json"
    sites = SitesInformation(data_file_path=str(local))

Defensive patterns

Strategy: fallback

Validate before calling

import socket
from urllib.parse import urlparse

def manifest_url_reachable(url: str, timeout: float = 5.0) -> bool:
    """Cheap DNS/TCP pre-check before constructing SitesInformation."""
    parsed = urlparse(url)
    try:
        socket.create_connection((parsed.hostname, parsed.port or 443), timeout=timeout).close()
        return True
    except OSError:
        return False

Try / catch

from sherlock_project.sites import SitesInformation

try:
    sites = SitesInformation()  # live manifest
except FileNotFoundError as err:
    # Note: FileNotFoundError here means network failure, not a missing local file.
    print(f"live manifest unreachable: {err}; using local snapshot")
    sites = SitesInformation(data_file_path="sherlock_project/resources/data.json")

Prevention

Cache a known-good copy of data.json locally and fall back to it when the live MANIFEST_URL is unreachable.
In CI/offline environments, always pass an explicit local --site path instead of relying on the network fetch.
Configure HTTPS_PROXY/REQUESTS_CA_BUNDLE up front in proxy or TLS-intercepting networks.
Treat FileNotFoundError from SitesInformation as a network error first — check connectivity before suspecting files.

When it happens

Trigger: Constructing SitesInformation() (or running the CLI) while offline; DNS failure for data.sherlockproject.xyz; a proxy/firewall blocking the egress request (requests raises ProxyError or ConnectTimeout); an SSL/TLS interception middlebox causing SSLError; a custom data_file_path that starts with 'http' but points at a dead host. The 30-second requests timeout also produces this if the server never responds.

Common situations: Corporate networks with mandatory proxies that reject the request; air-gapped or flaky connections; CI runners with restricted egress where data.sherlockproject.xyz is not on the allowlist; a typo'd custom --site URL (e.g. wrong hostname); the sherlock data host being temporarily down.

Related errors

AI-assisted analysis of sherlock-project/sherlock@9100f9d40a (2026-08-14). Data as JSON: /api/errors/8f29f8d68fe05ff3. Report an issue: GitHub.