tonhowtf/omniget · error · anyhow::Error

HTML request returned HTTP

Error message

HTML request returned HTTP {}

What it means

Thrown by TwitterDownloader::request_html_media when the HTTP response from the tweet HTML page has a non-success (non-2xx) status code. The HTML fallback strategy fetches the tweet page and scrapes pbs.twimg.com photo URLs out of it; this error means the page itself refused the request.

Solutions

1. Inspect the logged HTTP status: 404 means the tweet is gone, 403 means auth/blocking, 429 means rate limited.
2. Configure a valid, unexpired auth cookie so the request is authenticated.
3. Verify the tweet URL is public and contains media.
4. Add retry with backoff for 429/5xx statuses.
5. Fall back to an alternative extractor (yt-dlp) when the HTML strategy fails.

Example fix

// before
let response = request.send().await?;
if !response.status().is_success() {
    return Err(anyhow!("HTML request returned HTTP {}", response.status()));
}
// after: retry transient statuses before failing
let response = request.send().await?;
let status = response.status();
if status.as_u16() == 429 || status.is_server_error() {
    tokio::time::sleep(std::time::Duration::from_secs(2)).await;
    // re-issue request or delegate to another strategy
}
if !status.is_success() {
    return Err(anyhow!("HTML request returned HTTP {}", status));
}

// pre-flight: ensure the tweet page is reachable and we have auth
let status = reqwest::Client::new()
    .head(url).header("Referer", "https://x.com/").send().await?
    .status();
if !status.is_success() { eprintln!("tweet page pre-check failed: HTTP {}", status); }

fn is_retryable_status(status: u16) -> bool {
    matches!(status, 408 | 429) || (500..=599).contains(&status)
}

match request_html_media(&url).await {
    Err(e) if e.to_string().contains("HTTP 429") => {
        tokio::time::sleep(Duration::from_secs(30)).await;
        retry_limited(3, || request_html_media(&url)).await
    }
    Err(e) => Err(e),
    Ok(items) => Ok(items),
}

When it happens

Trigger: Calling get_media_info on a tweet whose GraphQL and syndication strategies already failed, causing request_html_media to GET the tweet URL with a Referer header (and optional auth cookie), and the server responding e.g. 404 (deleted tweet), 403 (blocked/age-gated), or 5xx.

Common situations: Deleted, suspended, or protected accounts; X requiring login for that tweet (no/expired auth cookie); rate limiting (429); corporate proxy or CDN blocking the request.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

• HTML request returned HTTP
• YouTube não retornou URL
• chunk de assinatura do X nao encontrado
• HTTP ao acessar pin
• HTTP ao acessar pin

AI-assisted analysis of tonhowtf/omniget@8600b91f42 (2026-09-12). Data as JSON: /api/errors/aa499c385e753b27. Report an issue: GitHub.

Appendix: source

Thrown at src-tauri/src/platforms/twitter/mod.rs:1048

        Ok(Self::media_info_from_twitter_media(
            filename_base,
            twitter_media,
        ))
    }

    async fn request_html_media(&self, url: &str) -> anyhow::Result<Vec<serde_json::Value>> {
        let mut request = self
            .client
            .get(url)
            .header("User-Agent", USER_AGENT)
            .header("Accept-Language", "en")
            .header("Referer", "https://x.com/");
        if let Some(cookie) = Self::auth_cookie_string() {
            request = request.header("Cookie", cookie);
        }
        let response = request.send().await?;
        if !response.status().is_success() {
            return Err(anyhow!("HTML request returned HTTP {}", response.status()));
        }
        let html = response.text().await?;
        let items = Self::extract_html_photo_items(&html);
        if items.is_empty() {
            return Err(anyhow!("No photo URLs found in HTML"));
        }
        tracing::debug!("[twitter] html extracted {} photo entries", items.len());
        Ok(items
            .into_iter()
            .map(|item| {
                serde_json::json!({
                    "type": "photo",
                    "media_url_https": item.url,
                })
            })
            .collect())
    }

View on GitHub (pinned to 8600b91f42)