crewAIInc/crewAI · error · ValueError

Invalid mode: {mode}. Must be either 'scrape' or 'crawl'

Error message

Invalid mode: {mode}. Must be either 'scrape' or 'crawl'

What it means

SpiderTool supports exactly two modes: 'scrape' (single page) and 'crawl' (site-wide up to DEFAULT_CRAWL_LIMIT). _run raises ValueError when the mode argument is anything else; the check is case-sensitive, so 'Scrape' or 'CRAWL' also fail.

Source

Thrown at lib/crewai-tools/src/crewai_tools/tools/spider_tool/spider_tool.py:175

            ValueError: If URL is invalid or missing, or if mode is invalid.
            ImportError: If spider-client package is not properly installed.
            ConnectionError: If network connection fails while accessing the URL.
            Exception: For other runtime errors.
        """
        try:
            params = {}
            url = website_url or self.website_url

            if not url:
                raise ValueError(
                    "Website URL must be provided either during initialization or execution"
                )

            if not self._validate_url(url):
                raise ValueError(f"Invalid URL format: {url}")

            if mode not in ["scrape", "crawl"]:
                raise ValueError(
                    f"Invalid mode: {mode}. Must be either 'scrape' or 'crawl'"
                )

            params = {
                "request": self.config.DEFAULT_REQUEST_MODE,
                "filter_output_svg": self.config.FILTER_SVG,
                "return_format": self.config.DEFAULT_RETURN_FORMAT,
            }

            if mode == "crawl":
                params["limit"] = self.config.DEFAULT_CRAWL_LIMIT

            if self.custom_params:
                params.update(self.custom_params)

            action = (
                self.spider.scrape_url if mode == "scrape" else self.spider.crawl_url
            )

View on GitHub (pinned to 754d7323be)

Solutions

1. Use exactly "scrape" or "crawl" (lowercase) for the mode argument.
2. Normalize incoming mode strings: mode.strip().lower() before calling _run.
3. Constrain agent-facing schemas to an enum of the two valid values so the model cannot emit others.

Example fix

# before
result = tool._run(website_url=url, mode="SCRAPE")

# after
result = tool._run(website_url=url, mode="scrape")

mode = (mode or "scrape").strip().lower()
if mode not in {"scrape", "crawl"}:
    raise ValueError(f"mode must be 'scrape' or 'crawl', got {mode!r}")

def is_spider_mode(value: str) -> bool:
    return isinstance(value, str) and value.strip().lower() in {"scrape", "crawl"}

try:
    result = tool._run(website_url=url, mode=mode)
except ValueError as e:
    if "Invalid mode" in str(e):
        result = tool._run(website_url=url, mode="scrape")  # safe default
    else:
        raise

When it happens

Trigger: Calling _run(mode="scraping"), mode="spider", or mode="SCRAPE" (capitalized); an LLM agent inventing mode values not in the tool schema; configuration files with a typo'd mode string.

Common situations: Agents producing free-form mode arguments; users assuming modes like 'screenshot' or 'search' exist because the Spider API supports them; casing inconsistencies between config and the tool's expectation.

Related errors

• Invalid data_type: '{raw_data_type}'. Valid values are: 'fil
• Query cannot be empty
• action={self.action!r} requires 'path'.
• action='append' requires 'content'. Pass the chunk to append
• action='move' requires 'destination'.

AI-assisted analysis of crewAIInc/crewAI@754d7323be (2026-08-15). Data as JSON: /api/errors/8ed298a0a866e66b. Report an issue: GitHub.