sigoden/aichat · warning · anyhow::Error

Invalid crawl page at

Error message

Invalid crawl page at {}

What it means

In `crawl_website`, discovered relative link paths are resolved against the normalized start URL with `Url::join`; when `join` fails for a path (malformed or otherwise unresolvable URL), the crawl task returns `anyhow!("Invalid crawl page at {}", path)`. The failure occurs before `crawl_page` runs, so the page is never fetched.

Solutions

1. Filter candidate links before enqueueing: keep only those that `Url::parse`/`join` successfully and whose scheme is http/https.
2. Sanitize hrefs on the crawled page (strip `javascript:`, `data:`, `mailto:` schemes) before starting the crawl.
3. Catch this error per-page and log a warning instead of failing the whole crawl.
4. If you control the source pages, fix the malformed links at the origin.

Example fix

// before
async move {
    let _permit = permit.await?;
    let url = normalized_start_url.join(&path)
        .map_err(|_| anyhow!("Invalid crawl page at {}", path))?;
    // ...
}
// after
async move {
    let Ok(url) = normalized_start_url.join(&path) else {
        eprintln!("skipping unparseable link: {path}");
        return Ok(None);
    };
    if !matches!(url.scheme(), "http" | "https") {
        return Ok(None);
    }
    // ...
}

fn crawlable(path: &str, base: &url::Url) -> Option<url::Url> {
    let url = base.join(path).ok()?;
    matches!(url.scheme(), "http" | "https").then_some(url)
}

// when driving crawl tasks yourself:
match tokio::join!(handle) {
    Err(e) if e.to_string().starts_with("Invalid crawl page at") => {
        log::warn!("skipped bad link: {e}");
    }
    other => other?,
}

When it happens

Trigger: The crawl discovers a link whose path cannot be joined onto `normalized_start_url` — e.g. a malformed href, a `data:`/`javascript:` pseudo-URL leaking into the path list, or an invalid percent-encoding.

Common situations: Crawling HTML pages with broken or exotic hrefs; scraping sites that emit `javascript:void(0)` or empty anchors as links; malformed sitemap entries.

Understand the failure class

Background: "Invalid URL" / "URL cannot be empty": fix the malformed or missing URL behind request-construction failures — this error's family across 50 libraries.

Related errors

• Invalid extract selector
• Invalid gh tree
• Invalid messages
• Failed to parse '.tools
• Unexpected media type

AI-assisted analysis of sigoden/aichat@82976d349a (2026-09-09). Data as JSON: /api/errors/b09d66c80e3734e4. Report an issue: GitHub.

Appendix: source

Thrown at src/utils/request.rs:250

    let mut result_pages = Vec::new();

    let mut index = 0;
    while index < paths.len() {
        let batch = paths[index..std::cmp::min(index + MAX_CRAWLS, paths.len())].to_vec();

        let tasks: Vec<_> = batch
            .iter()
            .map(|path| {
                let options = options.clone();
                let permit = semaphore.clone().acquire_owned(); // acquire a permit for concurrency control
                let normalized_start_url = normalized_start_url.clone();
                let path = path.clone();

                async move {
                    let _permit = permit.await?;
                    let url = normalized_start_url
                        .join(&path)
                        .map_err(|_| anyhow!("Invalid crawl page at {}", path))?;
                    let mut page = crawl_page(&normalized_start_url, &path, options)
                        .await
                        .with_context(|| format!("Failed to crawl {}", url.as_str()))?;
                    page.0 = url.as_str().to_string();
                    Ok(page)
                }
            })
            .collect();

        let results = stream::iter(tasks)
            .buffer_unordered(MAX_CRAWLS)
            .collect::<Vec<_>>()
            .await;

        let mut new_paths = Vec::new();

        for res in results {
            match res {

View on GitHub (pinned to 82976d349a)