{"record":{"id":"ed3e1d92abc500e9","repo":"unclecode/crawl4ai","slug":"invalid-source-s-valid-sources-are-joi","errorCode":null,"errorMessage":"Invalid source '{s}'. Valid sources are: {', '.join(valid_sources)}","messagePattern":"Invalid source '(.+?)'\\. Valid sources are: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"crawl4ai/async_url_seeder.py","lineNumber":409,"sourceCode":"        query = config.query\n        score_threshold = config.score_threshold\n        scoring_method = config.scoring_method\n\n        # Store cache config for use in _from_sitemaps\n        self._cache_ttl_hours = getattr(config, 'cache_ttl_hours', 24)\n        self._validate_sitemap_lastmod = getattr(config, 'validate_sitemap_lastmod', True)\n\n        # Ensure seeder's logger verbose matches the config's verbose if it's set\n        if self.logger and hasattr(self.logger, 'verbose') and config.verbose is not None:\n            self.logger.verbose = config.verbose\n\n        # Parse source parameter - split by '+' to get list of sources\n        sources = [s.strip().lower() for s in source.split(\"+\") if s.strip()]\n\n        valid_sources = {\"cc\", \"sitemap\"}\n        for s in sources:\n            if s not in valid_sources:\n                raise ValueError(\n                    f\"Invalid source '{s}'. Valid sources are: {', '.join(valid_sources)}\")\n\n            # ensure we have the latest CC collection id when the source is cc\n            if s == \"cc\" and self.index_id is None:\n                self.index_id = await self._latest_index()\n\n\n        if hits_per_sec:\n            if hits_per_sec <= 0:\n                self._log(\n                    \"warning\", \"hits_per_sec must be positive. Disabling rate limiting.\", tag=\"URL_SEED\")\n                self._rate_sem = None\n            else:\n                self._rate_sem = asyncio.Semaphore(hits_per_sec)\n        else:\n            self._rate_sem = None  # Ensure it's None if no rate limiting\n\n        self._log(\"info\", \"Starting URL seeding for {domain} with source={source}\",","sourceCodeStart":391,"sourceCodeEnd":427,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/crawl4ai/async_url_seeder.py#L391-L427","documentation":"AsyncUrlSeeder accepts a 'source' string made of '+'-separated tokens (e.g. 'cc+sitemap') and validates each lowercased token against {'cc', 'sitemap'}. Any token not in that set raises this ValueError before any network request is made. It is purely an input-validation error in the seeder API (urls()/many_urls() via config.source).","triggerScenarios":"Calling AsyncUrlSeeder().urls(source='commoncrawl') or passing SeedingConfig(source='CC, sitemap') — a comma-separated or misspelled source name. Also passing 'wayback', 'web', or any provider name the seeder does not support.","commonSituations":"Assuming the seeder supports more providers than it does (Wayback, Google); using ',' instead of '+' as the separator; typos or casing like 'CC ' handled but 'craw' not; version differences where only some sources exist.","solutions":["Use only the supported tokens: 'cc' (Common Crawl index) and 'sitemap', separated by '+': source='cc+sitemap'.","Check for typos, extra whitespace is trimmed but spelling must match exactly after lowering.","If you need other providers, fetch them separately; the seeder only ships cc and sitemap support."],"exampleFix":"# before\nurls = await seeder.urls('commoncrawl+sitemaps', ...)  # ValueError\n\n# after\nurls = await seeder.urls('cc+sitemap', ...)","handlingStrategy":"validation","validationCode":"VALID = {\"cc\", \"sitemap\"}\ndef normalize_source(source: str) -> str:\n    parts = [s.strip().lower() for s in source.split('+') if s.strip()]\n    bad = [s for s in parts if s not in VALID]\n    if bad:\n        raise ValueError(f'unsupported source(s): {bad}; valid: {sorted(VALID)}')\n    return '+'.join(parts)","typeGuard":"def is_valid_seeder_source(source: str) -> bool:\n    return all(p.strip().lower() in {\"cc\", \"sitemap\"} for p in source.split(\"+\") if p.strip())","tryCatchPattern":"try:\n    urls = await seeder.urls(source, ...)\nexcept ValueError as e:\n    if 'Invalid source' in str(e):\n        # log and fall back to a known-good source\n        urls = await seeder.urls('sitemap', ...)\n    else:\n        raise","preventionTips":["Centralize the source string in one validated config constant.","Use '+' as separator, never ','.","Assert is_valid_seeder_source(source) in tests for every config you ship."],"tags":["validation","url-seeder","input-validation"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}