{"record":{"id":"7ac23a6eeaae2ea9","repo":"unclecode/crawl4ai","slug":"invalid-url-make-sure-the-url-is-a-non-empty-stri","errorCode":null,"errorMessage":"Invalid URL, make sure the URL is a non-empty string","messagePattern":"Invalid URL, make sure the URL is a non-empty string","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"crawl4ai/async_webcrawler.py","lineNumber":252,"sourceCode":"            result = await crawler.arun(url=\"https://example.com\", config=config)\n\n        Args:\n            url: The URL to crawl (http://, https://, file://, or raw:)\n            config: Configuration object controlling crawl behavior\n            [other parameters maintained for backwards compatibility]\n\n        Returns:\n            CrawlResultContainer: A single-result container that proxies\n                attribute access to the underlying CrawlResult for backwards\n                compatibility (e.g. result.markdown, result.html).\n        \"\"\"\n        # Auto-start if not ready\n        if not self.ready:\n            await self.start()\n\n        config = config or CrawlerRunConfig()\n        if not isinstance(url, str) or not url:\n            raise ValueError(\n                \"Invalid URL, make sure the URL is a non-empty string\")\n\n        async with self._lock or self.nullcontext():\n            try:\n                self.logger.verbose = config.verbose\n\n                # Default to ENABLED if no cache mode specified\n                if config.cache_mode is None:\n                    config.cache_mode = CacheMode.ENABLED\n\n                # Create cache context\n                cache_context = CacheContext(url, config.cache_mode, False)\n\n                # Initialize processing variables\n                async_response: AsyncCrawlResponse = None\n                cached_result: CrawlResult = None\n                screenshot_data = None\n                pdf_data = None","sourceCodeStart":234,"sourceCodeEnd":270,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/crawl4ai/async_webcrawler.py#L234-L270","documentation":"AsyncWebCrawler.arun(url) requires url to be a non-empty str. The check runs after auto-start, before cache context creation, and rejects non-str types (None, bytes, ParseResult) and empty strings with this ValueError. It guards the very first step of a crawl.","triggerScenarios":"Calling arun(None) because a URL variable was never assigned; arun('') from an empty CSV/config cell; passing a yarl/urllib.parse.ParseResult or bytes object instead of a plain string; iterating a file whose lines are stripped to empty.","commonSituations":"Feeding crawler from data files where some rows lack URLs; loops that pass list elements of the wrong type; passing result of urlparse() directly; trailing whitespace-only strings still pass but '' does not.","solutions":["Ensure the value passed is a str: coerce with str(url) or extract .get('url') properly from your data source.","Skip empty/None URLs before calling arun: if not url or not isinstance(url, str): continue.","If you have a parsed URL object, convert it back with url.geturl() / str(url)."],"exampleFix":"# before\nfor row in rows:\n    result = await crawler.arun(row.get('url'))  # may be None\n\n# after\nfor row in rows:\n    url = row.get('url')\n    if not isinstance(url, str) or not url.strip():\n        continue\n    result = await crawler.arun(url)","handlingStrategy":"type-guard","validationCode":"def assert_url(url):\n    if not isinstance(url, str) or not url.strip():\n        raise ValueError(f'invalid url: {url!r}')\n    return url.strip()","typeGuard":"from typing import Any\ndef is_crawlable_url(url: Any) -> bool:\n    return isinstance(url, str) and bool(url.strip())","tryCatchPattern":"try:\n    result = await crawler.arun(url)\nexcept ValueError as e:\n    if 'Invalid URL' in str(e):\n        continue  # skip bad row in a batch loop\n    raise","preventionTips":["Filter URLs with is_crawlable_url before batches.","Type input sources as Iterable[str] in your own wrappers.","Strip and reject blank lines when reading URL lists from files."],"tags":["validation","webcrawler","input-validation"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}