What causes "Problem parsing json contents at '{data_file_path}': {error}." in sherlock-project/sherlock?

Fetch the URL manually (`curl -s | python -m json.tool`) — the exact decode error and the first offending bytes will show whether you got HTML instead of JSON.

sherlock-project/sherlock · error · ValueError

Problem parsing json contents at '{data_file_path}': {error

Error message

Problem parsing json contents at '{data_file_path}':  {error}.

What it means

The manifest URL returned HTTP 200, but response.json() failed to decode the body as JSON, so sites.py wraps the decode error (json.JSONDecodeError) in a ValueError. It means the bytes received were not a valid JSON document — typically an HTML error page, a login/captive-portal page, or a truncated body.

Source

Thrown at sherlock_project/sites.py:140

            data_file_path = MANIFEST_URL

        if data_file_path.lower().startswith("http"):
            # Reference is to a URL.
            try:
                response = requests.get(url=data_file_path, timeout=30)
            except Exception as error:
                raise FileNotFoundError(
                    f"Problem while attempting to access data file URL '{data_file_path}':  {error}"
                )

            if response.status_code != 200:
                raise FileNotFoundError(f"Bad response while accessing "
                                        f"data file URL '{data_file_path}'."
                                        )
            try:
                site_data = response.json()
            except Exception as error:
                raise ValueError(
                    f"Problem parsing json contents at '{data_file_path}':  {error}."
                )

        else:
            # Reference is to a file.
            try:
                with open(data_file_path, "r", encoding="utf-8") as file:
                    try:
                        site_data = json.load(file)
                    except Exception as error:
                        raise ValueError(
                            f"Problem parsing json contents at '{data_file_path}':  {error}."
                        )

            except FileNotFoundError:
                raise FileNotFoundError(f"Problem while attempting to access "
                                        f"data file '{data_file_path}'."
                                        )

View on GitHub (pinned to 9100f9d40a)

Solutions

Fetch the URL manually (`curl -s <url> | python -m json.tool`) — the exact decode error and the first offending bytes will show whether you got HTML instead of JSON.
If HTML came back, switch to the raw content URL (e.g. raw.githubusercontent.com for GitHub) or fix the server to serve the file with correct content.
Remove JSON-invalid syntax from a hand-edited manifest: BOMs, comments, trailing commas, single quotes; validate with `python -m json.tool data.json`.
For captive portals / proxies, complete authentication or bypass the network, then retry.

Example fix

# before
sites = SitesInformation(data_file_path="https://example.com/my-list")  # serves HTML

# after
import json, requests
r = requests.get("https://example.com/my-list.json", timeout=30)
json.loads(r.text)  # fail fast with the real JSON error before constructing SitesInformation
sites = SitesInformation(data_file_path="https://example.com/my-list.json")

Defensive patterns

Strategy: validation

Validate before calling

import json, requests

def fetch_manifest_json(url: str) -> dict:
    """Fetch and JSON-validate the manifest before handing it to sherlock."""
    resp = requests.get(url, timeout=30)
    resp.raise_for_status()
    try:
        return resp.json()
    except json.JSONDecodeError as err:
        snippet = resp.text[:120].replace("\n", " ")
        raise ValueError(f"Manifest at {url} is not JSON (starts with: {snippet!r})") from err

Try / catch

from sherlock_project.sites import SitesInformation

try:
    sites = SitesInformation(data_file_path=url)
except ValueError as err:
    # Covers JSON decode failure of a URL manifest; err names the path and cause.
    # Response was 200 but the body was not JSON (portal/block page/HTML) —
    # inspect the URL in a browser or curl, then retry or switch to a local file.
    raise

Prevention

Pre-flight the manifest with curl | python -m json.tool in setup scripts to catch HTML-instead-of-JSON early.
On public Wi-Fi, confirm internet access past captive portals before running sherlock.
Serve manifests with an explicit application/json content type and no BOM from your own hosts.
Keep a validated local snapshot for environments where middleboxes rewrite responses.

When it happens

Trigger: A captive portal or 'accept cookies' interstitial answering 200 with HTML; a CDN serving a block page (Cloudflare-style challenge) with status 200; a custom data_file_path URL that returns an HTML index page instead of the raw file (classic GitHub blob-vs-raw mistake); a man-in-the-middle proxy rewriting responses; a manifest actually formatted as JSON5/YAML or containing a BOM/comments.

Common situations: Public Wi-Fi captive portals; corporate TLS-termination proxies injecting notices; pointing --site at a web UI page rather than the raw file; hand-authored manifests saved with a UTF-8 BOM or trailing commas; a partially-written manifest uploaded mid-deploy.

Related errors

AI-assisted analysis of sherlock-project/sherlock@9100f9d40a (2026-08-14). Data as JSON: /api/errors/3c243e5c4e6758c2. Report an issue: GitHub.