crewAIInc/crewAI · error · ValueError

Both website_url and css_element must be provided.

Error message

Both website_url and css_element must be provided.

What it means

Raised by ScrapeElementFromWebsiteTool._run when either website_url or css_element ends up None. Values come from kwargs first, then the instance attributes set at construction; if the tool was created bare (no website_url) and the call omitted either key, this ValueError fires before any HTTP request.

Source

Thrown at lib/crewai-tools/src/crewai_tools/tools/scrape_element_from_website/scrape_element_from_website.py:83

            self.args_schema = FixedScrapeElementFromWebsiteToolSchema
            self._generate_description()
            if cookies is not None:
                self.cookies = {cookies["name"]: os.getenv(cookies["value"]) or ""}

    def _run(
        self,
        **kwargs: Any,
    ) -> Any:
        if not BEAUTIFULSOUP_AVAILABLE:
            raise ImportError(
                "beautifulsoup4 is not installed. Please install it with `pip install crewai-tools[beautifulsoup4]`"
            )

        website_url = kwargs.get("website_url", self.website_url)
        css_element = kwargs.get("css_element", self.css_element)

        if website_url is None or css_element is None:
            raise ValueError("Both website_url and css_element must be provided.")

        page = safe_get(
            website_url,
            headers=self.headers,
            cookies=self.cookies if self.cookies else {},
            timeout=30,
        )
        parsed = BeautifulSoup(page.content, "html.parser")
        elements = parsed.select(css_element)
        return "\n".join([element.get_text() for element in elements])

View on GitHub (pinned to 754d7323be)

Solutions

1. Pass both website_url and css_element explicitly to run(): tool.run(website_url='https://example.com', css_element='div.main')
2. Or set them at construction: ScrapeElementFromWebsiteTool(website_url='https://example.com', css_element='div.main')
3. Verify the exact expected kwarg names against FixedScrapeElementFromWebsiteToolSchema
4. When an agent drives the tool, make both fields required in the tool schema/description

Example fix

# before
tool = ScrapeElementFromWebsiteTool()
tool.run(website_url='https://example.com')  # css_element None -> ValueError

# after
tool.run(website_url='https://example.com', css_element='div.content')

def valid_scrape_args(kwargs: dict) -> bool:
    return bool(kwargs.get("website_url")) and bool(kwargs.get("css_element"))

assert valid_scrape_args(call_kwargs), "website_url and css_element are both required"

from typing import TypedDict

class ScrapeElementArgs(TypedDict, total=True):
    website_url: str
    css_element: str

try:
    tool.run(website_url=url, css_element=sel)
except ValueError as e:
    if "must be provided" in str(e):
        raise ValueError("ScrapeElementFromWebsiteTool requires both website_url and css_element") from e
    raise

When it happens

Trigger: ScrapeElementFromWebsiteTool() instantiated without website_url, then run with only one of website_url/css_element; or a schema mismatch where the agent/caller passes differently-named keys so kwargs.get returns None.

Common situations: LLM agents omitting the css_element argument because the tool description didn't make it mandatory; renaming of args between versions; constructing with website_url but calling with 'url' instead of 'website_url'.

Related errors

• Website URL must be provided.
• Project name '{name}' would generate folder name '{folder_na
• Project name '{name}' contains no valid characters for a Pyt
• Project name '{name}' would generate class name '{class_name
• Project name '{name}' would generate class name '{class_name

AI-assisted analysis of crewAIInc/crewAI@754d7323be (2026-08-15). Data as JSON: /api/errors/c20784f6731b879d. Report an issue: GitHub.