{"record":{"id":"ab06616f848e1a8d","repo":"unclecode/crawl4ai","slug":"invalid-source-s-invalid-valid-valid-source","errorCode":null,"errorMessage":"Invalid source(s): {invalid}. Valid: {VALID_SOURCES}","messagePattern":"Invalid source\\(s\\): (.+?)\\. Valid: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"crawl4ai/domain_mapper.py","lineNumber":235,"sourceCode":"            config: DomainMapperConfig. kwargs override config fields.\n\n        Returns:\n            List of dicts with url, host, source, status, head_data, relevance_score.\n        \"\"\"\n        from .async_configs import DomainMapperConfig as _Cfg\n        if config:\n            config = config.clone(**kwargs) if kwargs else config\n        else:\n            config = _Cfg(**kwargs) if kwargs else _Cfg()\n\n        if config.verbose is not None and self.logger:\n            self.logger.verbose = config.verbose\n\n        # Parse + validate sources\n        sources = {s.strip().lower() for s in config.source.split(\"+\") if s.strip()}\n        invalid = sources - VALID_SOURCES\n        if invalid:\n            raise ValueError(f\"Invalid source(s): {invalid}. Valid: {VALID_SOURCES}\")\n\n        # Rate limiter\n        if config.hits_per_sec and config.hits_per_sec > 0:\n            self._rate_sem = asyncio.Semaphore(config.hits_per_sec)\n        else:\n            self._rate_sem = None\n\n        # Normalize domain\n        base_domain = re.sub(r\"^https?://\", \"\", domain).strip(\"/\").lower()\n\n        self._log(\"info\", \"Scanning domain: {domain} with sources: {sources}\",\n                  params={\"domain\": base_domain, \"sources\": config.source})\n\n        # ── Phase 1: Host Discovery ──────────────────────────────────────\n        hosts = await self._discover_hosts(base_domain, sources, config)\n        self._log(\"info\", \"Discovered {count} live hosts\",\n                  params={\"count\": len(hosts)})\n","sourceCodeStart":217,"sourceCodeEnd":253,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/crawl4ai/domain_mapper.py#L217-L253","documentation":"Raised by DomainMapper.scan when the source string contains tokens outside VALID_SOURCES = {sitemap, cc, wayback, crt, probe, robots, feed, homepage}. Sources are parsed by splitting config.source on '+', lowercased and trimmed, then any unknown token triggers this ValueError listing both the invalid tokens and the valid set.","triggerScenarios":"Passing source=\"commoncrawl\" instead of \"cc\", source=\"common_crawl+dns\", or any string with a typo like \"sitemaps\" or \"robot\". Empty tokens from trailing '+' are filtered out, so only real misspelled tokens trigger it.","commonSituations":"Users writing the long-form service name (commoncrawl, common-crawl) rather than the abbreviation cc; guessing source names instead of checking the docs; casing/whitespace variants are tolerated but abbreviation mismatches are not.","solutions":["Use only the valid abbreviations joined with '+': 'sitemap+cc+wayback+crt+probe+robots+feed+homepage'","Replace commoncrawl with cc","Validate your source list against VALID_SOURCES (importable from crawl4ai.domain_mapper) before calling scan"],"exampleFix":"// before\nconfig = DomainMapperConfig(source=\"commoncrawl+dns\")\nawait mapper.scan(\"example.com\", config=config)  # ValueError\n\n// after\nconfig = DomainMapperConfig(source=\"cc+probe\")\nawait mapper.scan(\"example.com\", config=config)","handlingStrategy":"validation","validationCode":"from crawl4ai.domain_mapper import VALID_SOURCES\n\nsources = {s.strip().lower() for s in source_str.split(\"+\") if s.strip()}\ninvalid = sources - VALID_SOURCES\nif invalid:\n    raise ValueError(f\"fix these sources before scanning: {invalid}\")","typeGuard":"from crawl4ai.domain_mapper import VALID_SOURCES\n\ndef is_valid_source_string(s: str) -> bool:\n    parts = {p.strip().lower() for p in s.split(\"+\") if p.strip()}\n    return bool(parts) and parts <= VALID_SOURCES","tryCatchPattern":null,"preventionTips":["Use only the abbreviations: sitemap, cc, wayback, crt, probe, robots, feed, homepage","Remember 'commoncrawl' is 'cc'","Validate against VALID_SOURCES before scan() when sources come from user input"],"tags":["domain-mapper","validation","config"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}