unclecode/crawl4ai · error · ValueError

Invalid source(s): {invalid}. Valid: {VALID_SOURCES}

Error message

Invalid source(s): {invalid}. Valid: {VALID_SOURCES}

What it means

Raised by DomainMapper.scan when the source string contains tokens outside VALID_SOURCES = {sitemap, cc, wayback, crt, probe, robots, feed, homepage}. Sources are parsed by splitting config.source on '+', lowercased and trimmed, then any unknown token triggers this ValueError listing both the invalid tokens and the valid set.

Source

Thrown at crawl4ai/domain_mapper.py:235

            config: DomainMapperConfig. kwargs override config fields.

        Returns:
            List of dicts with url, host, source, status, head_data, relevance_score.
        """
        from .async_configs import DomainMapperConfig as _Cfg
        if config:
            config = config.clone(**kwargs) if kwargs else config
        else:
            config = _Cfg(**kwargs) if kwargs else _Cfg()

        if config.verbose is not None and self.logger:
            self.logger.verbose = config.verbose

        # Parse + validate sources
        sources = {s.strip().lower() for s in config.source.split("+") if s.strip()}
        invalid = sources - VALID_SOURCES
        if invalid:
            raise ValueError(f"Invalid source(s): {invalid}. Valid: {VALID_SOURCES}")

        # Rate limiter
        if config.hits_per_sec and config.hits_per_sec > 0:
            self._rate_sem = asyncio.Semaphore(config.hits_per_sec)
        else:
            self._rate_sem = None

        # Normalize domain
        base_domain = re.sub(r"^https?://", "", domain).strip("/").lower()

        self._log("info", "Scanning domain: {domain} with sources: {sources}",
                  params={"domain": base_domain, "sources": config.source})

        # ── Phase 1: Host Discovery ──────────────────────────────────────
        hosts = await self._discover_hosts(base_domain, sources, config)
        self._log("info", "Discovered {count} live hosts",
                  params={"count": len(hosts)})

View on GitHub (pinned to 7e80152142)

Solutions

1. Use only the valid abbreviations joined with '+': 'sitemap+cc+wayback+crt+probe+robots+feed+homepage'
2. Replace commoncrawl with cc
3. Validate your source list against VALID_SOURCES (importable from crawl4ai.domain_mapper) before calling scan

Example fix

// before
config = DomainMapperConfig(source="commoncrawl+dns")
await mapper.scan("example.com", config=config)  # ValueError

// after
config = DomainMapperConfig(source="cc+probe")
await mapper.scan("example.com", config=config)

from crawl4ai.domain_mapper import VALID_SOURCES

sources = {s.strip().lower() for s in source_str.split("+") if s.strip()}
invalid = sources - VALID_SOURCES
if invalid:
    raise ValueError(f"fix these sources before scanning: {invalid}")

from crawl4ai.domain_mapper import VALID_SOURCES

def is_valid_source_string(s: str) -> bool:
    parts = {p.strip().lower() for p in s.split("+") if p.strip()}
    return bool(parts) and parts <= VALID_SOURCES

When it happens

Trigger: Passing source="commoncrawl" instead of "cc", source="common_crawl+dns", or any string with a typo like "sitemaps" or "robot". Empty tokens from trailing '+' are filtered out, so only real misspelled tokens trigger it.

Common situations: Users writing the long-form service name (commoncrawl, common-crawl) rather than the abbreviation cc; guessing source names instead of checking the docs; casing/whitespace variants are tolerated but abbreviation mismatches are not.

Related errors

• too many hooks (max 10)
• unknown hook action {action!r}; allowed: {sorted(HOOK_REGIST
• invalid params for hook '{action}': {e}
• str(e)
• Rejected config: {e}

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/ab06616f848e1a8d. Report an issue: GitHub.