{"record":{"id":"861733685023f897","repo":"sherlock-project/sherlock","slug":"problem-parsing-json-contents-at-data-file-path-861733","errorCode":null,"errorMessage":"Problem parsing json contents at '{data_file_path}':  Missing attribute {error}.","messagePattern":"Problem parsing json contents at '(.+?)':  Missing attribute (.+?)\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sherlock_project/sites.py","lineNumber":200,"sourceCode":"                honor_exclusions = False\n\n        self.sites = {}\n\n        # Add all site information from the json file to internal site list.\n        for site_name in site_data:\n            try:\n\n                self.sites[site_name] = \\\n                    SiteInformation(site_name,\n                                    site_data[site_name][\"urlMain\"],\n                                    site_data[site_name][\"url\"],\n                                    site_data[site_name][\"username_claimed\"],\n                                    site_data[site_name],\n                                    site_data[site_name].get(\"isNSFW\",False)\n\n                                    )\n            except KeyError as error:\n                raise ValueError(\n                    f\"Problem parsing json contents at '{data_file_path}':  Missing attribute {error}.\"\n                )\n            except TypeError:\n                print(f\"Encountered TypeError parsing json contents for target '{site_name}' at {data_file_path}\\nSkipping target.\\n\")\n\n        return\n\n    def remove_nsfw_sites(self, do_not_remove: list = []):\n        \"\"\"\n        Remove NSFW sites from the sites, if isNSFW flag is true for site\n\n        Keyword Arguments:\n        self                   -- This object.\n\n        Return Value:\n        None\n        \"\"\"\n        sites = {}","sourceCodeStart":182,"sourceCodeEnd":218,"githubUrl":"https://github.com/sherlock-project/sherlock/blob/9100f9d40a3274bd46f4ce903c5c6fee6f3745bc/sherlock_project/sites.py#L182-L218","documentation":"After loading the manifest, sites.py constructs a SiteInformation for every site entry, indexing the required keys urlMain, url and username_claimed. A missing key raises KeyError, which is caught and re-raised as ValueError naming the offending attribute. The schema check is per-site and strict: every entry must carry all three required attributes.","triggerScenarios":"A custom/edited data.json where a site entry omits \"urlMain\" (most common), \"url\", or \"username_claimed\"; entries generated by scripts that only emit url + errorType; renaming a key (e.g. \"mainUrl\" instead of \"urlMain\") in a fork's manifest; upstream manifest schema drift when using a very old sherlock against a newer manifest (or vice versa). The message names the exact missing key, e.g. \"Missing attribute 'urlMain'.\".","commonSituations":"Contributing a new site to data.json and forgetting username_claimed (needed for sherlock's self-tests); validating a third-party manifest against a stricter sherlock version that now requires urlMain; programmatically built manifests missing fields; merging manifests where a partial entry survived a failed merge.","solutions":["Read the attribute name from the message and add it to the named site entry: every site needs urlMain (site homepage), url (profile URL template with {} for the username) and username_claimed (a username known to exist).","Cross-check against a known-good entry in the official data.json and against the schema section ($schema) at the top of the manifest.","Run the repo's manifest tests after editing (sherlock's test suite validates every site via its username_claimed) to catch omissions before runtime.","If a schema change upstream caused it, align versions: update sherlock to match the manifest generation you are loading, or pin the manifest to the one shipped with your sherlock version."],"exampleFix":"// data.json site entry\n// before\n\"ExampleSite\": {\n  \"url\": \"https://example.com/{}\",\n  \"errorType\": \"status_code\"\n}\n// -> ValueError: Missing attribute 'urlMain'.\n\n// after\n\"ExampleSite\": {\n  \"url\": \"https://example.com/{}\",\n  \"urlMain\": \"https://example.com\",\n  \"username_claimed\": \"blue\",\n  \"errorType\": \"status_code\"\n}","handlingStrategy":"validation","validationCode":"import json\n\nREQUIRED_SITE_KEYS = {\"url\", \"urlMain\", \"username_claimed\"}\n\ndef validate_manifest_schema(path: str) -> dict:\n    with open(path, \"r\", encoding=\"utf-8\") as f:\n        data = json.load(f)\n    data.pop(\"$schema\", None)\n    bad = [\n        name for name, info in data.items()\n        if not isinstance(info, dict) or not REQUIRED_SITE_KEYS.issubset(info)\n    ]\n    if bad:\n        raise ValueError(f\"Site entries missing required keys {REQUIRED_SITE_KEYS}: {bad}\")\n    return data","typeGuard":"REQUIRED_SITE_KEYS = {\"url\", \"urlMain\", \"username_claimed\"}\n\ndef is_valid_site_entry(site_info) -> bool:\n    \"\"\"True when the entry carries all keys SiteInformation() requires.\"\"\"\n    return (\n        isinstance(site_info, dict)\n        and REQUIRED_SITE_KEYS.issubset(site_info)\n        and isinstance(site_info[\"url\"], str)\n        and isinstance(site_info[\"urlMain\"], str)\n        and isinstance(site_info[\"username_claimed\"], str)\n    )","tryCatchPattern":"from sherlock_project.sites import SitesInformation\n\ntry:\n    sites = SitesInformation(data_file_path=path)\nexcept ValueError as err:\n    if \"Missing attribute\" in str(err):\n        # err names the exact key (e.g. 'urlMain') and the manifest path;\n        # add the key to that site entry, or drop the broken entry, then retry.\n        ...\n    raise","preventionTips":["Require every new site entry to include url, urlMain and username_claimed; review with a schema lint before commit.","Use the message's attribute name to jump straight to the missing field instead of eyeballing the whole file.","When loading third-party manifests, pre-filter with an is_valid_site_entry() guard and skip bad entries rather than aborting.","Keep sherlock and its manifest generation in sync — schema drift between versions shows up as suddenly-missing attributes."],"tags":["sherlock","manifest","schema","keyerror","data-validation"],"backgroundTag":null,"analyzedSha":"9100f9d40a3274bd46f4ce903c5c6fee6f3745bc","analyzedAt":"2026-08-14T19:48:26.535Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}