{"record":{"id":"8228c1f88e0f2a5d","repo":"ScrapeGraphAI/Scrapegraph-ai","slug":"unsupported-backend-self-backend","errorCode":null,"errorMessage":"Unsupported backend: {self.backend}","messagePattern":"Unsupported backend: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"warning","filePath":"scrapegraphai/docloaders/chromium.py","lineNumber":110,"sourceCode":"        self.urls = urls\n        self.load_state = load_state\n        self.requires_js_support = requires_js_support\n        self.storage_state = storage_state\n        self.backend = kwargs.get(\"backend\", backend)\n        self.browser_name = kwargs.get(\"browser_name\", browser_name)\n        self.retry_limit = kwargs.get(\"retry_limit\", retry_limit)\n        self.timeout = kwargs.get(\"timeout\", timeout)\n\n    async def scrape(self, url: str) -> str:\n        if self.backend == \"playwright\":\n            return await self.ascrape_playwright(url)\n        elif self.backend == \"selenium\":\n            try:\n                return await self.ascrape_undetected_chromedriver(url)\n            except Exception as e:\n                raise ValueError(f\"Failed to scrape with undetected chromedriver: {e}\")\n        else:\n            raise ValueError(f\"Unsupported backend: {self.backend}\")\n\n    async def ascrape_undetected_chromedriver(self, url: str) -> str:\n        \"\"\"\n        Asynchronously scrape the content of a given URL using undetected chrome with Selenium.\n\n        Args:\n            url (str): The URL to scrape.\n\n        Returns:\n            str: The scraped HTML content or an error message if an exception occurs.\n        \"\"\"\n        try:\n            import undetected_chromedriver as uc\n        except ImportError:\n            raise ImportError(\n                \"undetected_chromedriver is required for ChromiumLoader. Please install it with `pip install undetected-chromedriver`.\"\n            )\n","sourceCodeStart":92,"sourceCodeEnd":128,"githubUrl":"https://github.com/ScrapeGraphAI/Scrapegraph-ai/blob/532dfffbf6ee823a6c9cf8cfedc24a93bf026780/scrapegraphai/docloaders/chromium.py#L92-L128","documentation":"After fetching and parsing robots.txt, RobotsNode asks an LLM whether scraping is allowed; if the answer contains 'no' and force_scraping is not enabled, it raises ValueError('The website you selected is not scrapable'). This is a deliberate compliance guard honoring the site's robots.txt disallow rules.","triggerScenarios":"Running a graph with robots compliance enabled against a site whose robots.txt disallows the path/user-agent; force_scraping defaults to False so any 'no' verdict raises.","commonSituations":"Targeting sites that block bots in robots.txt; enabling the robots-check graph variant without realizing it enforces compliance; testing scrapers against protected or paywalled domains.","solutions":["Respect the site's policy: choose a different target or the site's official API.","If you have authorization, set force_scraping=True in the config to bypass the check (it will only log a warning).","Check https://<site>/robots.txt yourself first to confirm the disallow rule applies to your path and user-agent."],"exampleFix":"# before\nconfig = {\"force_scraping\": False}\n\n# after (only if you are permitted to scrape)\nconfig = {\"force_scraping\": True}","handlingStrategy":"try-catch","validationCode":"# optional: check robots.txt yourself first\nfrom urllib.robotparser import RobotFileParser\nrp = RobotFileParser()\nrp.set_url(\"https://example.com/robots.txt\")\nrp.read()\nif not rp.can_fetch(\"*\", target_url):\n    # pick another target or set force_scraping knowingly\n    ...","typeGuard":"null","tryCatchPattern":"try:\n    result = graph.run()\nexcept ValueError as e:\n    if \"not scrapable\" in str(e):\n        # site disallows scraping; choose another source or API instead of forcing\n        ...","preventionTips":["Check the target's robots.txt before running large scrape jobs.","Keep force_scraping=False by default and only enable it when you are authorized to scrape.","Prefer official APIs for sites that disallow bots."],"tags":["robots-txt","compliance","scraping","blocked"],"backgroundTag":"robots-txt-disallowed","analyzedSha":"532dfffbf6ee823a6c9cf8cfedc24a93bf026780","analyzedAt":"2026-08-28T15:19:38.821Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}