{"record":{"id":"b09d66c80e3734e4","repo":"sigoden/aichat","slug":"invalid-crawl-page-at","errorCode":null,"errorMessage":"Invalid crawl page at {}","messagePattern":"Invalid crawl page at (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"warning","filePath":"src/utils/request.rs","lineNumber":250,"sourceCode":"    let mut result_pages = Vec::new();\n\n    let mut index = 0;\n    while index < paths.len() {\n        let batch = paths[index..std::cmp::min(index + MAX_CRAWLS, paths.len())].to_vec();\n\n        let tasks: Vec<_> = batch\n            .iter()\n            .map(|path| {\n                let options = options.clone();\n                let permit = semaphore.clone().acquire_owned(); // acquire a permit for concurrency control\n                let normalized_start_url = normalized_start_url.clone();\n                let path = path.clone();\n\n                async move {\n                    let _permit = permit.await?;\n                    let url = normalized_start_url\n                        .join(&path)\n                        .map_err(|_| anyhow!(\"Invalid crawl page at {}\", path))?;\n                    let mut page = crawl_page(&normalized_start_url, &path, options)\n                        .await\n                        .with_context(|| format!(\"Failed to crawl {}\", url.as_str()))?;\n                    page.0 = url.as_str().to_string();\n                    Ok(page)\n                }\n            })\n            .collect();\n\n        let results = stream::iter(tasks)\n            .buffer_unordered(MAX_CRAWLS)\n            .collect::<Vec<_>>()\n            .await;\n\n        let mut new_paths = Vec::new();\n\n        for res in results {\n            match res {","sourceCodeStart":232,"sourceCodeEnd":268,"githubUrl":"https://github.com/sigoden/aichat/blob/82976d349ad97ac9aae0655ad631dace5e2a6385/src/utils/request.rs#L232-L268","documentation":"In `crawl_website`, discovered relative link paths are resolved against the normalized start URL with `Url::join`; when `join` fails for a path (malformed or otherwise unresolvable URL), the crawl task returns `anyhow!(\"Invalid crawl page at {}\", path)`. The failure occurs before `crawl_page` runs, so the page is never fetched.","triggerScenarios":"The crawl discovers a link whose path cannot be joined onto `normalized_start_url` — e.g. a malformed href, a `data:`/`javascript:` pseudo-URL leaking into the path list, or an invalid percent-encoding.","commonSituations":"Crawling HTML pages with broken or exotic hrefs; scraping sites that emit `javascript:void(0)` or empty anchors as links; malformed sitemap entries.","solutions":["Filter candidate links before enqueueing: keep only those that `Url::parse`/`join` successfully and whose scheme is http/https.","Sanitize hrefs on the crawled page (strip `javascript:`, `data:`, `mailto:` schemes) before starting the crawl.","Catch this error per-page and log a warning instead of failing the whole crawl.","If you control the source pages, fix the malformed links at the origin."],"exampleFix":"// before\nasync move {\n    let _permit = permit.await?;\n    let url = normalized_start_url.join(&path)\n        .map_err(|_| anyhow!(\"Invalid crawl page at {}\", path))?;\n    // ...\n}\n// after\nasync move {\n    let Ok(url) = normalized_start_url.join(&path) else {\n        eprintln!(\"skipping unparseable link: {path}\");\n        return Ok(None);\n    };\n    if !matches!(url.scheme(), \"http\" | \"https\") {\n        return Ok(None);\n    }\n    // ...\n}","handlingStrategy":"validation","validationCode":"fn crawlable(path: &str, base: &url::Url) -> Option<url::Url> {\n    let url = base.join(path).ok()?;\n    matches!(url.scheme(), \"http\" | \"https\").then_some(url)\n}","typeGuard":null,"tryCatchPattern":"// when driving crawl tasks yourself:\nmatch tokio::join!(handle) {\n    Err(e) if e.to_string().starts_with(\"Invalid crawl page at\") => {\n        log::warn!(\"skipped bad link: {e}\");\n    }\n    other => other?,\n}","preventionTips":["Filter hrefs to http/https schemes before enqueueing crawl pages.","Drop javascript:, data:, mailto:, and empty anchors during link extraction.","Pre-resolve every candidate URL with Url::join and skip failures.","Fix malformed links at the source site where you control it."],"tags":["crawler","url","html","validation"],"backgroundTag":"invalid-url","analyzedSha":"82976d349ad97ac9aae0655ad631dace5e2a6385","analyzedAt":"2026-09-09T18:33:06.139Z","contentChangedAt":"2026-09-09T18:33:06.139Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}