unclecode/crawl4ai · error · InvalidArgumentException

Failed to crawl {url}: {e.msg}

Error message

Failed to crawl {url}: {e.msg}

What it means

Error "Failed to crawl {url}: {e.msg}" thrown in unclecode/crawl4ai.

Source

Thrown at crawl4ai/legacy/crawler_strategy.py:319

            # Store in cache
            cache_file_path = os.path.join(
                os.getenv("CRAWL4_AI_BASE_DIRECTORY", Path.home()),
                ".crawl4ai",
                "cache",
                url_hash,
            )
            with open(cache_file_path, "w", encoding="utf-8") as f:
                f.write(html)

            if self.verbose:
                print(f"[LOG] ✅ Crawled {url} successfully!")

            return html
        except InvalidArgumentException as e:
            if not hasattr(e, "msg"):
                e.msg = sanitize_input_encode(str(e))
            raise InvalidArgumentException(f"Failed to crawl {url}: {e.msg}")
        except WebDriverException as e:
            # If e does nlt have msg attribute create it and set it to str(e)
            if not hasattr(e, "msg"):
                e.msg = sanitize_input_encode(str(e))
            raise WebDriverException(f"Failed to crawl {url}: {e.msg}")
        except Exception as e:
            if not hasattr(e, "msg"):
                e.msg = sanitize_input_encode(str(e))
            raise Exception(f"Failed to crawl {url}: {e.msg}")

    def take_screenshot(self) -> str:
        try:
            # Get the dimensions of the page
            total_width = self.driver.execute_script("return document.body.scrollWidth")
            total_height = self.driver.execute_script(
                "return document.body.scrollHeight"
            )

View on GitHub (pinned to 7e80152142)

Solutions

1. Inspect the underlying WebDriver exception message for the root cause (network, timeout, driver crash).
2. Retry the crawl and ensure the Selenium driver and browser versions are compatible.

Example fix

strategy.crawl(url)  # wrap in try/except and log e.msg

When it happens

Trigger: Thrown at crawl4ai/legacy/crawler_strategy.py:319 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/333767d15edcc009. Report an issue: GitHub.