{"record":{"id":"3aef347113d279cd","repo":"tonhowtf/omniget","slug":"pagina-html-vazia","errorCode":null,"errorMessage":"pagina HTML vazia","messagePattern":"pagina HTML vazia","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src-tauri/omniget-core/src/core/tools/arxiv.rs","lineNumber":1238,"sourceCode":"async fn fetch_source(client: &reqwest::Client, r: &ArxivRef) -> anyhow::Result<SourceBundle> {\n    let resp = client.get(r.source_url()).send().await?;\n    if !resp.status().is_success() {\n        return Err(anyhow!(\"HTTP {}\", resp.status()));\n    }\n    let bytes = resp.bytes().await?;\n    extract_source(&bytes)\n}\n\nasync fn fetch_html(client: &reqwest::Client, r: &ArxivRef) -> anyhow::Result<String> {\n    let resp = client.get(r.html_url()).send().await?;\n    if !resp.status().is_success() {\n        return Err(anyhow!(\"HTTP {}\", resp.status()));\n    }\n    let html = resp.text().await?;\n    let html = mathml_to_tex(&html);\n    let md = htmd::convert(&html).map_err(|e| anyhow!(\"HTML para Markdown: {}\", e))?;\n    if md.trim().is_empty() {\n        return Err(anyhow!(\"pagina HTML vazia\"));\n    }\n    Ok(md)\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn reconhece_as_formas_de_id() {\n        let cases = [\n            (\"2401.12345\", \"2401.12345\", None),\n            (\"2401.12345v2\", \"2401.12345\", Some(2)),\n            (\"arXiv:2401.12345\", \"2401.12345\", None),\n            (\"arxiv.org/abs/2401.12345\", \"2401.12345\", None),\n            (\"https://arxiv.org/abs/2401.12345v3\", \"2401.12345\", Some(3)),\n            (\n                \"https://arxiv.org/pdf/2401.12345v1.pdf\",","sourceCodeStart":1220,"sourceCodeEnd":1256,"githubUrl":"https://github.com/tonhowtf/omniget/blob/8600b91f4246848bac346874daa9e61c1fc5677a/src-tauri/omniget-core/src/core/tools/arxiv.rs#L1220-L1256","documentation":"fetch_html rejects the result when htmd's Markdown output is entirely whitespace, returning \"pagina HTML vazia\" (empty HTML page). This guards against treating a technically-successful conversion of an empty/useless page as a valid paper text.","triggerScenarios":"fetch -> fetch_html where the downloaded HTML contains no convertible body content: consent/error interstitials, pages that are only scripts/styles, or papers whose HTML rendition is an empty shell.","commonSituations":"arXiv HTML page that is a redirect/notice page; cloudflare or maintenance interstitial served with 200; content behind JavaScript not present in raw HTML.","solutions":["Fall back to fetch_source (LaTeX) extraction when HTML yields empty Markdown","Log/inspect the raw HTML when this fires to detect interstitial pages","Check Content-Type and response length before converting","Consider parsing the PDF as a last resort"],"exampleFix":"// before\nif md.trim().is_empty() {\n    return Err(anyhow!(\"pagina HTML vazia\"));\n}\n// after\nif md.trim().is_empty() {\n    tracing::warn!(\"HTML rendition empty for {}, falling back to source\", r.id);\n    return fetch_source(client, r).and_then(|_| Ok(String::new())); // wire real fallback\n}","handlingStrategy":"fallback","validationCode":"// Cheap pre-check: ensure the response body has real content\nanyhow::ensure!(html.len() > 512, \"HTML page too small ({} bytes); likely an interstitial\", html.len());","typeGuard":null,"tryCatchPattern":"match fetch_html(&client, &r).await {\n    Ok(md) if !md.trim().is_empty() => md,\n    _ => fetch_source_text(&client, &r).await?, // fallback to LaTeX extraction\n}","preventionTips":["Check Content-Type is text/html before parsing","Detect interstitial/notice pages by markers before conversion","Always provide a secondary extraction path","Log raw HTML snippets when conversion output is empty"],"tags":["html","markdown","empty-content","rust"],"backgroundTag":"empty-result-set","analyzedSha":"8600b91f4246848bac346874daa9e61c1fc5677a","analyzedAt":"2026-09-12T14:29:19.317Z","contentChangedAt":"2026-09-12T14:29:19.317Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}