{"record":{"id":"7b483d8089dfe15f","repo":"ScrapeGraphAI/Scrapegraph-ai","slug":"failed-to-scrape-with-undetected-chromedriver-e","errorCode":null,"errorMessage":"Failed to scrape with undetected chromedriver: {e}","messagePattern":"Failed to scrape with undetected chromedriver: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"scrapegraphai/docloaders/chromium.py","lineNumber":108,"sourceCode":"        self.headless = headless\n        self.proxy = parse_or_search_proxy(proxy) if proxy else None\n        self.urls = urls\n        self.load_state = load_state\n        self.requires_js_support = requires_js_support\n        self.storage_state = storage_state\n        self.backend = kwargs.get(\"backend\", backend)\n        self.browser_name = kwargs.get(\"browser_name\", browser_name)\n        self.retry_limit = kwargs.get(\"retry_limit\", retry_limit)\n        self.timeout = kwargs.get(\"timeout\", timeout)\n\n    async def scrape(self, url: str) -> str:\n        if self.backend == \"playwright\":\n            return await self.ascrape_playwright(url)\n        elif self.backend == \"selenium\":\n            try:\n                return await self.ascrape_undetected_chromedriver(url)\n            except Exception as e:\n                raise ValueError(f\"Failed to scrape with undetected chromedriver: {e}\")\n        else:\n            raise ValueError(f\"Unsupported backend: {self.backend}\")\n\n    async def ascrape_undetected_chromedriver(self, url: str) -> str:\n        \"\"\"\n        Asynchronously scrape the content of a given URL using undetected chrome with Selenium.\n\n        Args:\n            url (str): The URL to scrape.\n\n        Returns:\n            str: The scraped HTML content or an error message if an exception occurs.\n        \"\"\"\n        try:\n            import undetected_chromedriver as uc\n        except ImportError:\n            raise ImportError(\n                \"undetected_chromedriver is required for ChromiumLoader. Please install it with `pip install undetected-chromedriver`.\"","sourceCodeStart":90,"sourceCodeEnd":126,"githubUrl":"https://github.com/ScrapeGraphAI/Scrapegraph-ai/blob/532dfffbf6ee823a6c9cf8cfedc24a93bf026780/scrapegraphai/docloaders/chromium.py#L90-L126","documentation":"RobotsNode.execute raises ValueError('Operation not allowed') when the source it receives does not start with 'http'. The node fetches {base_url}/robots.txt, so it can only operate on http(s) URLs; local files or raw strings are rejected upfront.","triggerScenarios":"Passing a local file path ('./page.html'), an empty string, or a bare domain ('example.com') as the source to a graph containing RobotsNode; an upstream node producing a non-URL source.","commonSituations":"Switching a graph from a web source to a local file input while keeping robots checking enabled; forgetting the https:// scheme when building URLs dynamically.","solutions":["Provide a full http(s) URL as the source (e.g. 'https://example.com/page').","If you want to scrape local files, remove RobotsNode from the graph.","Normalize inputs upstream: prepend 'https://' when the scheme is missing."],"exampleFix":"# before\nsource = \"example.com/article\"\n\n# after\nsource = \"https://example.com/article\"","handlingStrategy":"validation","validationCode":"from urllib.parse import urlparse\nif not source.startswith((\"http://\", \"https://\")):\n    raise ValueError(\"RobotsNode requires an http(s) URL source\")","typeGuard":"def is_http_url(source: str) -> bool:\n    return isinstance(source, str) and urlparse(source).scheme in (\"http\", \"https\")","tryCatchPattern":"try:\n    result = graph.run()\nexcept ValueError as e:\n    if \"Operation not allowed\" in str(e):\n        # normalize the source URL and retry\n        ...","preventionTips":["Normalize sources to absolute https:// URLs before feeding graphs with RobotsNode.","Remove RobotsNode when scraping local files."],"tags":["robots-txt","input-validation","url","source"],"backgroundTag":"invalid-url-format","analyzedSha":"532dfffbf6ee823a6c9cf8cfedc24a93bf026780","analyzedAt":"2026-08-28T15:19:38.821Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}