{"record":{"id":"c5c9992187fbbf19","repo":"crewAIInc/crewAI","slug":"website-url-must-be-provided-either-during-initial-c5c999","errorCode":null,"errorMessage":"Website URL must be provided either during initialization or execution","messagePattern":"Website URL must be provided either during initialization or execution","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"lib/crewai-tools/src/crewai_tools/tools/spider_tool/spider_tool.py","lineNumber":167,"sourceCode":"                - \"scrape\": Extract content from single page\n                - \"crawl\": Follow links and extract content from multiple pages\n\n        Returns:\n            Optional[str]: Extracted content in markdown format, or None if extraction fails\n                        and log_failures is True.\n\n        Raises:\n            ValueError: If URL is invalid or missing, or if mode is invalid.\n            ImportError: If spider-client package is not properly installed.\n            ConnectionError: If network connection fails while accessing the URL.\n            Exception: For other runtime errors.\n        \"\"\"\n        try:\n            params = {}\n            url = website_url or self.website_url\n\n            if not url:\n                raise ValueError(\n                    \"Website URL must be provided either during initialization or execution\"\n                )\n\n            if not self._validate_url(url):\n                raise ValueError(f\"Invalid URL format: {url}\")\n\n            if mode not in [\"scrape\", \"crawl\"]:\n                raise ValueError(\n                    f\"Invalid mode: {mode}. Must be either 'scrape' or 'crawl'\"\n                )\n\n            params = {\n                \"request\": self.config.DEFAULT_REQUEST_MODE,\n                \"filter_output_svg\": self.config.FILTER_SVG,\n                \"return_format\": self.config.DEFAULT_RETURN_FORMAT,\n            }\n\n            if mode == \"crawl\":","sourceCodeStart":149,"sourceCodeEnd":185,"githubUrl":"https://github.com/crewAIInc/crewAI/blob/754d7323beb2fd042e33444a115ea2d5a47193f0/lib/crewai-tools/src/crewai_tools/tools/spider_tool/spider_tool.py#L149-L185","documentation":"SpiderTool._run requires a target website: it resolves website_url from the _run argument, falling back to self.website_url set at construction. If both are empty/None it raises ValueError telling you the URL must be supplied at initialization or execution. All of the tool's modes (scrape/crawl) operate on a URL, so there is no meaningful default.","triggerScenarios":"Creating SpiderTool() without website_url and calling _run() with no website_url argument; passing website_url=\"\" or None both places; an agent tool-call that omits the url parameter when the tool had no default.","commonSituations":"Agents expected to pass the URL at runtime but generating calls without it; configuration loading where the URL env var/setting is empty; reusable tool instances shared across tasks with no fixed target.","solutions":["Pass the URL at call time: tool._run(website_url=\"https://example.com\", mode=\"scrape\").","Or fix it for all calls: SpiderTool(website_url=\"https://example.com\").","For agent usage, ensure the tool input schema marks the URL required (or set a default) so the model always supplies it."],"exampleFix":"# before\ntool = SpiderTool(api_key=...)\nresult = tool._run()\n\n# after\nresult = tool._run(website_url=\"https://example.com\", mode=\"scrape\")","handlingStrategy":"validation","validationCode":"url = (website_url or default_url or \"\").strip()\nif not url:\n    raise ValueError(\"website_url is required: pass it to SpiderTool(...) or _run(website_url=...)\")\nresult = tool._run(website_url=url, mode=mode)","typeGuard":"def has_spider_target(tool_url: str | None, call_url: str | None) -> bool:\n    return bool((call_url or tool_url or \"\").strip())","tryCatchPattern":"try:\n    result = tool._run(website_url=url, mode=mode)\nexcept ValueError as e:\n    if \"Website URL must be provided\" in str(e):\n        return \"No URL supplied; provide a website_url.\"  # agent re-ask\n    raise","preventionTips":["Set website_url at construction for single-site tools, or enforce a required url field in agent schemas.","Strip and check the URL string before invoking the tool."],"tags":["spider","input-validation","required-argument"],"backgroundTag":null,"analyzedSha":"754d7323beb2fd042e33444a115ea2d5a47193f0","analyzedAt":"2026-08-15T04:06:56.746Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}