sherlock-project/sherlock · error · ValueError

Problem parsing json contents at '{data_file_path}': Missin

Error message

Problem parsing json contents at '{data_file_path}':  Missing attribute {error}.

What it means

After loading the manifest, sites.py constructs a SiteInformation for every site entry, indexing the required keys urlMain, url and username_claimed. A missing key raises KeyError, which is caught and re-raised as ValueError naming the offending attribute. The schema check is per-site and strict: every entry must carry all three required attributes.

Source

Thrown at sherlock_project/sites.py:200

                honor_exclusions = False

        self.sites = {}

        # Add all site information from the json file to internal site list.
        for site_name in site_data:
            try:

                self.sites[site_name] = \
                    SiteInformation(site_name,
                                    site_data[site_name]["urlMain"],
                                    site_data[site_name]["url"],
                                    site_data[site_name]["username_claimed"],
                                    site_data[site_name],
                                    site_data[site_name].get("isNSFW",False)

                                    )
            except KeyError as error:
                raise ValueError(
                    f"Problem parsing json contents at '{data_file_path}':  Missing attribute {error}."
                )
            except TypeError:
                print(f"Encountered TypeError parsing json contents for target '{site_name}' at {data_file_path}\nSkipping target.\n")

        return

    def remove_nsfw_sites(self, do_not_remove: list = []):
        """
        Remove NSFW sites from the sites, if isNSFW flag is true for site

        Keyword Arguments:
        self                   -- This object.

        Return Value:
        None
        """
        sites = {}

View on GitHub (pinned to 9100f9d40a)

Solutions

Read the attribute name from the message and add it to the named site entry: every site needs urlMain (site homepage), url (profile URL template with {} for the username) and username_claimed (a username known to exist).
Cross-check against a known-good entry in the official data.json and against the schema section ($schema) at the top of the manifest.
Run the repo's manifest tests after editing (sherlock's test suite validates every site via its username_claimed) to catch omissions before runtime.
If a schema change upstream caused it, align versions: update sherlock to match the manifest generation you are loading, or pin the manifest to the one shipped with your sherlock version.

Example fix

// data.json site entry
// before
"ExampleSite": {
  "url": "https://example.com/{}",
  "errorType": "status_code"
}
// -> ValueError: Missing attribute 'urlMain'.

// after
"ExampleSite": {
  "url": "https://example.com/{}",
  "urlMain": "https://example.com",
  "username_claimed": "blue",
  "errorType": "status_code"
}

Defensive patterns

Strategy: validation

Validate before calling

import json

REQUIRED_SITE_KEYS = {"url", "urlMain", "username_claimed"}

def validate_manifest_schema(path: str) -> dict:
    with open(path, "r", encoding="utf-8") as f:
        data = json.load(f)
    data.pop("$schema", None)
    bad = [
        name for name, info in data.items()
        if not isinstance(info, dict) or not REQUIRED_SITE_KEYS.issubset(info)
    ]
    if bad:
        raise ValueError(f"Site entries missing required keys {REQUIRED_SITE_KEYS}: {bad}")
    return data

Type guard

REQUIRED_SITE_KEYS = {"url", "urlMain", "username_claimed"}

def is_valid_site_entry(site_info) -> bool:
    """True when the entry carries all keys SiteInformation() requires."""
    return (
        isinstance(site_info, dict)
        and REQUIRED_SITE_KEYS.issubset(site_info)
        and isinstance(site_info["url"], str)
        and isinstance(site_info["urlMain"], str)
        and isinstance(site_info["username_claimed"], str)
    )

Try / catch

from sherlock_project.sites import SitesInformation

try:
    sites = SitesInformation(data_file_path=path)
except ValueError as err:
    if "Missing attribute" in str(err):
        # err names the exact key (e.g. 'urlMain') and the manifest path;
        # add the key to that site entry, or drop the broken entry, then retry.
        ...
    raise

Prevention

Require every new site entry to include url, urlMain and username_claimed; review with a schema lint before commit.
Use the message's attribute name to jump straight to the missing field instead of eyeballing the whole file.
When loading third-party manifests, pre-filter with an is_valid_site_entry() guard and skip bad entries rather than aborting.
Keep sherlock and its manifest generation in sync — schema drift between versions shows up as suddenly-missing attributes.

When it happens

Trigger: A custom/edited data.json where a site entry omits "urlMain" (most common), "url", or "username_claimed"; entries generated by scripts that only emit url + errorType; renaming a key (e.g. "mainUrl" instead of "urlMain") in a fork's manifest; upstream manifest schema drift when using a very old sherlock against a newer manifest (or vice versa). The message names the exact missing key, e.g. "Missing attribute 'urlMain'.".

Common situations: Contributing a new site to data.json and forgetting username_claimed (needed for sherlock's self-tests); validating a third-party manifest against a stricter sherlock version that now requires urlMain; programmatically built manifests missing fields; merging manifests where a partial entry survived a failed merge.

Related errors

AI-assisted analysis of sherlock-project/sherlock@9100f9d40a (2026-08-14). Data as JSON: /api/errors/861733685023f897. Report an issue: GitHub.