unclecode/crawl4ai · error · HTTPException

Deep crawling with stream currently supports exactly one URL

Error message

Deep crawling with stream currently supports exactly one URL per request. Received {len(urls)} URLs.

What it means

Error "Deep crawling with stream currently supports exactly one URL per request. Received {len(urls)} URLs." thrown in unclecode/crawl4ai.

Source

Thrown at deploy/docker/api.py:881

    try:
        # SSRF guard: validate every seed URL's destination before fetching,
        # mirroring handle_crawl_request. The streaming path previously skipped
        # this, leaving /crawl/stream (and /crawl with stream=true) unguarded.
        urls = _normalize_and_validate_seeds(urls)
        browser_config = BrowserConfig.load(browser_config, provenance=Provenance.UNTRUSTED)
        # browser_config.verbose = True # Set to False or remove for production stress testing
        browser_config.verbose = False
        from egress_broker import enforce_egress
        enforce_egress(browser_config)
        crawler_config = CrawlerRunConfig.load(crawler_config, provenance=Provenance.UNTRUSTED)
        from governor import clamp_deep_crawl
        clamp_deep_crawl(crawler_config)
        crawler_config.scraping_strategy = LXMLWebScrapingStrategy()
        crawler_config.stream = True

        # Deep crawl streaming supports exactly one start URL
        if crawler_config.deep_crawl_strategy is not None and len(urls) != 1:
            raise HTTPException(
                status_code=status.HTTP_400_BAD_REQUEST,
                detail=(
                    "Deep crawling with stream currently supports exactly one URL per request. "
                    f"Received {len(urls)} URLs."
                ),
            )

        from crawler_pool import get_crawler, release_crawler
        crawler = await get_crawler(browser_config)

        # Attach declarative hooks if provided
        if hooks_config:
            hooks_status = _attach_declarative_hooks(crawler, hooks_config)
            logger.info(f"Hooks attachment status for streaming: {hooks_status['status']}")
            hooks_info = {'status': hooks_status}

        # Deep crawl with single URL: use arun() which returns an async generator
        # mirroring the Python library's streaming behavior

View on GitHub (pinned to 7e80152142)

Solutions

1. Send exactly one URL per request when using deep crawling with stream=true.
2. Disable streaming or split multiple URLs into separate requests.

Example fix

POST /crawl {"urls": ["https://example.com"], "deep_crawl": true, "stream": true}

When it happens

Trigger: Thrown at deploy/docker/api.py:881 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/6632343ba9ce90b2. Report an issue: GitHub.