ScrapeGraphAI/Scrapegraph-ai · error · ValueError

Failed to scrape with undetected chromedriver: {e}

Error message

Failed to scrape with undetected chromedriver: {e}

What it means

RobotsNode.execute raises ValueError('Operation not allowed') when the source it receives does not start with 'http'. The node fetches {base_url}/robots.txt, so it can only operate on http(s) URLs; local files or raw strings are rejected upfront.

Source

Thrown at scrapegraphai/docloaders/chromium.py:108

        self.headless = headless
        self.proxy = parse_or_search_proxy(proxy) if proxy else None
        self.urls = urls
        self.load_state = load_state
        self.requires_js_support = requires_js_support
        self.storage_state = storage_state
        self.backend = kwargs.get("backend", backend)
        self.browser_name = kwargs.get("browser_name", browser_name)
        self.retry_limit = kwargs.get("retry_limit", retry_limit)
        self.timeout = kwargs.get("timeout", timeout)

    async def scrape(self, url: str) -> str:
        if self.backend == "playwright":
            return await self.ascrape_playwright(url)
        elif self.backend == "selenium":
            try:
                return await self.ascrape_undetected_chromedriver(url)
            except Exception as e:
                raise ValueError(f"Failed to scrape with undetected chromedriver: {e}")
        else:
            raise ValueError(f"Unsupported backend: {self.backend}")

    async def ascrape_undetected_chromedriver(self, url: str) -> str:
        """
        Asynchronously scrape the content of a given URL using undetected chrome with Selenium.

        Args:
            url (str): The URL to scrape.

        Returns:
            str: The scraped HTML content or an error message if an exception occurs.
        """
        try:
            import undetected_chromedriver as uc
        except ImportError:
            raise ImportError(
                "undetected_chromedriver is required for ChromiumLoader. Please install it with `pip install undetected-chromedriver`."

View on GitHub (pinned to 532dfffbf6)

Solutions

1. Provide a full http(s) URL as the source (e.g. 'https://example.com/page').
2. If you want to scrape local files, remove RobotsNode from the graph.
3. Normalize inputs upstream: prepend 'https://' when the scheme is missing.

Example fix

# before
source = "example.com/article"

# after
source = "https://example.com/article"

from urllib.parse import urlparse
if not source.startswith(("http://", "https://")):
    raise ValueError("RobotsNode requires an http(s) URL source")

def is_http_url(source: str) -> bool:
    return isinstance(source, str) and urlparse(source).scheme in ("http", "https")

try:
    result = graph.run()
except ValueError as e:
    if "Operation not allowed" in str(e):
        # normalize the source URL and retry
        ...

When it happens

Trigger: Passing a local file path ('./page.html'), an empty string, or a bare domain ('example.com') as the source to a graph containing RobotsNode; an upstream node producing a non-URL source.

Common situations: Switching a graph from a web source to a local file input while keeping robots checking enabled; forgetting the https:// scheme when building URLs dynamically.

Related errors

• Unsupported backend: {self.backend}
• {self.node_name} requires at least {self.min_input_len} inpu

AI-assisted analysis of ScrapeGraphAI/Scrapegraph-ai@532dfffbf6 (2026-08-28). Data as JSON: /api/errors/7b483d8089dfe15f. Report an issue: GitHub.