ScrapeGraphAI/Scrapegraph-ai · warning · ValueError
Unsupported backend: {self.backend}
Error message
Unsupported backend: {self.backend} What it means
After fetching and parsing robots.txt, RobotsNode asks an LLM whether scraping is allowed; if the answer contains 'no' and force_scraping is not enabled, it raises ValueError('The website you selected is not scrapable'). This is a deliberate compliance guard honoring the site's robots.txt disallow rules.
Source
Thrown at scrapegraphai/docloaders/chromium.py:110
self.urls = urls
self.load_state = load_state
self.requires_js_support = requires_js_support
self.storage_state = storage_state
self.backend = kwargs.get("backend", backend)
self.browser_name = kwargs.get("browser_name", browser_name)
self.retry_limit = kwargs.get("retry_limit", retry_limit)
self.timeout = kwargs.get("timeout", timeout)
async def scrape(self, url: str) -> str:
if self.backend == "playwright":
return await self.ascrape_playwright(url)
elif self.backend == "selenium":
try:
return await self.ascrape_undetected_chromedriver(url)
except Exception as e:
raise ValueError(f"Failed to scrape with undetected chromedriver: {e}")
else:
raise ValueError(f"Unsupported backend: {self.backend}")
async def ascrape_undetected_chromedriver(self, url: str) -> str:
"""
Asynchronously scrape the content of a given URL using undetected chrome with Selenium.
Args:
url (str): The URL to scrape.
Returns:
str: The scraped HTML content or an error message if an exception occurs.
"""
try:
import undetected_chromedriver as uc
except ImportError:
raise ImportError(
"undetected_chromedriver is required for ChromiumLoader. Please install it with `pip install undetected-chromedriver`."
)
View on GitHub (pinned to 532dfffbf6)
Solutions
- Respect the site's policy: choose a different target or the site's official API.
- If you have authorization, set force_scraping=True in the config to bypass the check (it will only log a warning).
- Check https://<site>/robots.txt yourself first to confirm the disallow rule applies to your path and user-agent.
Example fix
# before
config = {"force_scraping": False}
# after (only if you are permitted to scrape)
config = {"force_scraping": True} Defensive patterns
Strategy: try-catch
Validate before calling
# optional: check robots.txt yourself first
from urllib.robotparser import RobotFileParser
rp = RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()
if not rp.can_fetch("*", target_url):
# pick another target or set force_scraping knowingly
... Type guard
null
Try / catch
try:
result = graph.run()
except ValueError as e:
if "not scrapable" in str(e):
# site disallows scraping; choose another source or API instead of forcing
... Prevention
- Check the target's robots.txt before running large scrape jobs.
- Keep force_scraping=False by default and only enable it when you are authorized to scrape.
- Prefer official APIs for sites that disallow bots.
When it happens
Trigger: Running a graph with robots compliance enabled against a site whose robots.txt disallows the path/user-agent; force_scraping defaults to False so any 'no' verdict raises.
Common situations: Targeting sites that block bots in robots.txt; enabling the robots-check graph variant without realizing it enforces compliance; testing scrapers against protected or paywalled domains.
Related errors
AI-assisted analysis of ScrapeGraphAI/Scrapegraph-ai@532dfffbf6 (2026-08-28).
Data as JSON: /api/errors/8228c1f88e0f2a5d.
Report an issue: GitHub.