unclecode/crawl4ai · warning · Error

Container not found: ${config.container_selector}

Error message

Container not found: ${config.container_selector}

What it means

A 400 Bad Request returned by the streaming crawl endpoint when the request enables a deep crawl strategy (`deep_crawl_strategy` is set on crawler_config) but supplies more (or fewer) than one start URL. Deep-crawl streaming expands one start URL into a tree of links, so the implementation only supports a single seed per streamed request.

Source

Thrown at crawl4ai/async_crawler_strategy.py:1324

            # Import VirtualScrollConfig to avoid circular import
            from .async_configs import VirtualScrollConfig
            
            # Ensure config is a VirtualScrollConfig instance
            if isinstance(config, dict):
                config = VirtualScrollConfig.from_dict(config)
            
            self.logger.info(
                message="Starting virtual scroll capture for container: {selector}",
                tag="VSCROLL",
                params={"selector": config.container_selector}
            )
            
            # JavaScript function to handle virtual scroll capture
            virtual_scroll_js = """
            async (config) => {
                const container = document.querySelector(config.container_selector);
                if (!container) {
                    throw new Error(`Container not found: ${config.container_selector}`);
                }
                
                // List to store HTML chunks when content is replaced
                const htmlChunks = [];
                let previousHTML = container.innerHTML;
                let scrollCount = 0;
                
                // Determine scroll amount
                let scrollAmount;
                if (typeof config.scroll_by === 'number') {
                    scrollAmount = config.scroll_by;
                } else if (config.scroll_by === 'page_height') {
                    scrollAmount = window.innerHeight;
                } else { // container_height
                    scrollAmount = container.offsetHeight;
                }
                
                // Perform scrolling

View on GitHub (pinned to 7e80152142)

Solutions

  1. Send exactly one start URL per streaming deep-crawl request
  2. For multiple URLs, either issue one stream request per URL or drop deep_crawl_strategy and use the plain streaming batch mode
  3. For non-streaming deep crawl of several seeds, use the non-stream endpoint if your deployment supports it there

Example fix

# before
await stream_crawl(urls=["https://a.com", "https://b.com"],
                    crawler_config={"deep_crawl_strategy": BFSDeepCrawlStrategy()})

# after
for url in ["https://a.com", "https://b.com"]:
    async for ev in stream_crawl(urls=[url],
                                 crawler_config={"deep_crawl_strategy": BFSDeepCrawlStrategy()}):
        ...
Defensive patterns

Strategy: validation

Validate before calling

def validate_stream_request(urls, crawler_config):
    if crawler_config.get("deep_crawl_strategy"):
        if len(urls) != 1:
            raise ValueError("deep-crawl streaming requires exactly 1 URL")
    return True

Try / catch

try:
    async for ev in stream_crawl(urls, crawler_config): ...
except HTTPException as e:
    if e.status_code == 400 and "exactly one URL" in e.detail:
        for u in urls:  # fall back to one stream per URL
            async for ev in stream_crawl([u], crawler_config): ...
    else:
        raise

Prevention

When it happens

Trigger: POSTing to the stream endpoint with crawler_config containing a deep_crawl_strategy (e.g. BFSDeepCrawlStrategy) while `urls` in the payload has 0 or 2+ entries. The check `len(urls) != 1` fires before any browser is acquired.

Common situations: Reusing a batch-crawl payload (many URLs) against /crawl/stream after adding deep_crawl settings; enabling deep crawl in shared config and forgetting the endpoint takes exactly one seed URL; passing an empty urls list.

Related errors

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/07d8f7204212e79a. Report an issue: GitHub.