unclecode/crawl4ai · error · ValueError

Invalid source '{s}'. Valid sources are: {', '.join(valid_so

Error message

Invalid source '{s}'. Valid sources are: {', '.join(valid_sources)}

What it means

AsyncUrlSeeder accepts a 'source' string made of '+'-separated tokens (e.g. 'cc+sitemap') and validates each lowercased token against {'cc', 'sitemap'}. Any token not in that set raises this ValueError before any network request is made. It is purely an input-validation error in the seeder API (urls()/many_urls() via config.source).

Source

Thrown at crawl4ai/async_url_seeder.py:409

        query = config.query
        score_threshold = config.score_threshold
        scoring_method = config.scoring_method

        # Store cache config for use in _from_sitemaps
        self._cache_ttl_hours = getattr(config, 'cache_ttl_hours', 24)
        self._validate_sitemap_lastmod = getattr(config, 'validate_sitemap_lastmod', True)

        # Ensure seeder's logger verbose matches the config's verbose if it's set
        if self.logger and hasattr(self.logger, 'verbose') and config.verbose is not None:
            self.logger.verbose = config.verbose

        # Parse source parameter - split by '+' to get list of sources
        sources = [s.strip().lower() for s in source.split("+") if s.strip()]

        valid_sources = {"cc", "sitemap"}
        for s in sources:
            if s not in valid_sources:
                raise ValueError(
                    f"Invalid source '{s}'. Valid sources are: {', '.join(valid_sources)}")

            # ensure we have the latest CC collection id when the source is cc
            if s == "cc" and self.index_id is None:
                self.index_id = await self._latest_index()


        if hits_per_sec:
            if hits_per_sec <= 0:
                self._log(
                    "warning", "hits_per_sec must be positive. Disabling rate limiting.", tag="URL_SEED")
                self._rate_sem = None
            else:
                self._rate_sem = asyncio.Semaphore(hits_per_sec)
        else:
            self._rate_sem = None  # Ensure it's None if no rate limiting

        self._log("info", "Starting URL seeding for {domain} with source={source}",

View on GitHub (pinned to 7e80152142)

Solutions

1. Use only the supported tokens: 'cc' (Common Crawl index) and 'sitemap', separated by '+': source='cc+sitemap'.
2. Check for typos, extra whitespace is trimmed but spelling must match exactly after lowering.
3. If you need other providers, fetch them separately; the seeder only ships cc and sitemap support.

Example fix

# before
urls = await seeder.urls('commoncrawl+sitemaps', ...)  # ValueError

# after
urls = await seeder.urls('cc+sitemap', ...)

VALID = {"cc", "sitemap"}
def normalize_source(source: str) -> str:
    parts = [s.strip().lower() for s in source.split('+') if s.strip()]
    bad = [s for s in parts if s not in VALID]
    if bad:
        raise ValueError(f'unsupported source(s): {bad}; valid: {sorted(VALID)}')
    return '+'.join(parts)

def is_valid_seeder_source(source: str) -> bool:
    return all(p.strip().lower() in {"cc", "sitemap"} for p in source.split("+") if p.strip())

try:
    urls = await seeder.urls(source, ...)
except ValueError as e:
    if 'Invalid source' in str(e):
        # log and fall back to a known-good source
        urls = await seeder.urls('sitemap', ...)
    else:
        raise

When it happens

Trigger: Calling AsyncUrlSeeder().urls(source='commoncrawl') or passing SeedingConfig(source='CC, sitemap') — a comma-separated or misspelled source name. Also passing 'wayback', 'web', or any provider name the seeder does not support.

Common situations: Assuming the seeder supports more providers than it does (Wayback, Google); using ',' instead of '+' as the separator; typos or casing like 'CC ' handled but 'craw' not; version differences where only some sources exist.

Related errors

• `domain_or_domains` must be a string or a list of strings.
• Invalid URL, make sure the URL is a non-empty string
• Container not found: ${config.container_selector}
• Timeout after {timeout}ms waiting for selector '{wait_for}'
• [NSTProxy] token and channel_id are required

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/ed3e1d92abc500e9. Report an issue: GitHub.