tonhowtf/omniget · error
HTTP
Error message
HTTP {} What it means
arXiv fetch_source downloads the paper's LaTeX/e-print source bundle via reqwest and fails fast with anyhow!("HTTP {}", status) whenever the response status is not a success (2xx). This is a deliberate early-exit so the caller never tries to extract a source archive from an error page. It surfaces the raw HTTP status code (e.g. 404, 503) as the error message.
Solutions
- Check the arXiv ID/ref is valid and the paper actually has an e-print source (test https://arxiv.org/e-print/<id> in a browser)
- Implement retry with backoff for 429/503, honoring the Retry-After header
- Log resp.status() and fall back to fetch_html or PDF download when source fetch fails
- Verify network/proxy configuration allows reaching export.arxiv.org
Example fix
// before
let resp = client.get(r.source_url()).send().await?;
if !resp.status().is_success() {
return Err(anyhow!("HTTP {}", resp.status()));
}
// after
let resp = client.get(r.source_url()).send().await?;
if resp.status() == reqwest::StatusCode::TOO_MANY_REQUESTS || resp.status() == reqwest::StatusCode::SERVICE_UNAVAILABLE {
tokio::time::sleep(Duration::from_secs(3)).await;
return fetch_source(client, r).await; // retry
}
if !resp.status().is_success() {
anyhow::bail!("arXiv source fetch failed ({}): {}", resp.status(), r.source_url());
} Defensive patterns
Strategy: retry
Validate before calling
// Rust: pre-check reachability of the e-print URL before the real call
let head = client.head(r.source_url()).send().await?;
anyhow::ensure!(head.status().is_success(), "arXiv source unavailable: {}", head.status()); Try / catch
match fetch_source(&client, &r).await {
Ok(bundle) => bundle,
Err(e) if e.to_string().contains("HTTP 5") || e.to_string().contains("HTTP 429") => {
tokio::time::sleep(Duration::from_secs(5)).await;
fetch_source(&client, &r).await?
}
Err(e) => return Err(e),
} Prevention
- Validate arXiv IDs before fetching
- Add exponential backoff for 429/503 from arXiv
- Respect arXiv rate limits (delay between requests)
- Always have a fallback path (HTML or PDF extraction)
When it happens
Trigger: Calling fetch -> fetch_source with an ArxivRef whose source_url() returns a non-2xx response: arXiv ID does not exist, the paper has no e-print source (scanned PDF-only submissions), rate limiting (429/503 from arXiv), or network proxies returning 4xx/5xx.
Common situations: Typo in arXiv ID (404); arXiv API throttling during bulk fetching (503 with Retry-After); withdrawn papers; very old papers whose source is unavailable; corporate proxy blocking export.arxiv.org.
Understand the failure class
Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.
Related errors
AI-assisted analysis of tonhowtf/omniget@8600b91f42 (2026-09-12).
Data as JSON: /api/errors/9ac4378e0e517a84.
Report an issue: GitHub.
Appendix: source
Thrown at src-tauri/omniget-core/src/core/tools/arxiv.rs:1223
};
report(&p, ID, "done", 3, Some(3), None);
Ok(ArxivDoc {
chars: markdown.chars().count() as u64,
math_blocks: count_math(&markdown),
meta,
markdown,
body_source: body_source.to_string(),
path,
source_files: files,
fallback_reason: fallback,
})
}
async fn fetch_source(client: &reqwest::Client, r: &ArxivRef) -> anyhow::Result<SourceBundle> {
let resp = client.get(r.source_url()).send().await?;
if !resp.status().is_success() {
return Err(anyhow!("HTTP {}", resp.status()));
}
let bytes = resp.bytes().await?;
extract_source(&bytes)
}
async fn fetch_html(client: &reqwest::Client, r: &ArxivRef) -> anyhow::Result<String> {
let resp = client.get(r.html_url()).send().await?;
if !resp.status().is_success() {
return Err(anyhow!("HTTP {}", resp.status()));
}
let html = resp.text().await?;
let html = mathml_to_tex(&html);
let md = htmd::convert(&html).map_err(|e| anyhow!("HTML para Markdown: {}", e))?;
if md.trim().is_empty() {
return Err(anyhow!("pagina HTML vazia"));
}
Ok(md)
}View on GitHub (pinned to 8600b91f42)