tonhowtf/omniget · error

HTML request returned HTTP {}

Error message

HTML request returned HTTP {}

What it means

In `request_html_media` (src-tauri/omniget-core/src/platforms/twitter.rs:981), the Twitter platform fetches a tweet's HTML page as a fallback when the API path is unavailable. After sending the request (with Referer and optional Cookie headers), any non-success HTTP status causes this error, carrying the status code in the message. It exists so the caller (`native_get_media_info`) gets a clear signal that the HTML fallback scrape could not even retrieve the page, distinct from a parse failure.

Solutions

1. Log the HTTP status from the error message and handle it: refresh or correct the auth cookie if 401/403, back off and retry if 429, verify the tweet URL exists if 404.
2. Re-authenticate: update the cookie used by Self::auth_cookie_string() with a fresh logged-in session's cookie string.
3. Verify the tweet URL is public and still live by opening it in a browser (or curl with the same headers) before retrying.
4. Add retry-with-backoff around request_html_media for transient 5xx/429 statuses.
5. Check network egress (proxy/VPN/firewall) if the same request succeeds from a browser.

Example fix

// before: single attempt, no status-specific handling
let response = request.send().await?;
if !response.status().is_success() {
    return Err(anyhow!("HTML request returned HTTP {}", response.status()));
}

// after: refresh cookie and retry once on auth failure
let response = request.send().await?;
if response.status() == reqwest::StatusCode::UNAUTHORIZED
    || response.status() == reqwest::StatusCode::FORBIDDEN
{
    Self::refresh_auth_cookie().await; // re-login / rotate cookie
    let response = request.header("Cookie", Self::auth_cookie_string().unwrap_or_default())
        .send().await?;
}
if !response.status().is_success() {
    return Err(anyhow!("HTML request returned HTTP {}", response.status()));
}

// Rust: probe the tweet URL before invoking the library
let client = reqwest::Client::new();
let probe = client.head(url)
    .header("Referer", "https://x.com/")
    .send().await?;
if !probe.status().is_success() {
    anyhow::bail!("precheck failed: tweet page returned {}", probe.status());
}

fn is_http_ok(status: u16) -> bool { (200..300).contains(&status) }

match native_get_media_info(url).await {
    Ok(info) => use_media(info),
    Err(e) if e.to_string().contains("HTML request returned HTTP 429") => {
        tokio::time::sleep(Duration::from_secs(30)).await;
        retry_with_backoff(url, 3).await
    }
    Err(e) if e.to_string().contains("401") || e.to_string().contains("403") => {
        refresh_twitter_cookie(); // re-auth then retry once
        retry_once(url).await
    }
    Err(e) => { log::error!("twitter fetch failed: {e}"); show_user_error(e); }
}

When it happens

Trigger: Any call to native_get_media_info for a Twitter/X photo URL where the underlying reqwest request to x.com returns 4xx or 5xx: tweet deleted or made private, account suspended, auth cookie expired/invalid, rate limiting (HTTP 429), or Cloudflare/proxy blocking the request.

Common situations: Expired or malformed cookies in the stored auth (Self::auth_cookie_string()), scraping a protected/deleted tweet, X.com serving 403/429 to non-browser traffic, corporate proxy or geo-blocking, or X changing endpoints so old URLs 404.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

• HTML request returned HTTP {}
• YouTube não retornou URL
• chunk de assinatura do X nao encontrado
• HTTP {} ao acessar pin {}
• HTTP {} ao acessar pin {}

AI-assisted analysis of tonhowtf/omniget@8600b91f42 (2026-09-12). Data as JSON: /api/errors/6be7a811734432ec. Report an issue: GitHub.

Appendix: source

Thrown at src-tauri/omniget-core/src/platforms/twitter.rs:981

        Ok(Self::media_info_from_twitter_media(
            filename_base,
            twitter_media,
        ))
    }

    async fn request_html_media(&self, url: &str) -> anyhow::Result<Vec<serde_json::Value>> {
        let mut request = self
            .client
            .get(url)
            .header("User-Agent", USER_AGENT)
            .header("Accept-Language", "en")
            .header("Referer", "https://x.com/");
        if let Some(cookie) = Self::auth_cookie_string() {
            request = request.header("Cookie", cookie);
        }
        let response = request.send().await?;
        if !response.status().is_success() {
            return Err(anyhow!("HTML request returned HTTP {}", response.status()));
        }
        let html = response.text().await?;
        let items = Self::extract_html_photo_items(&html);
        if items.is_empty() {
            return Err(anyhow!("No photo URLs found in HTML"));
        }
        tracing::debug!("[twitter] html extracted {} photo entries", items.len());
        Ok(items
            .into_iter()
            .map(|item| {
                serde_json::json!({
                    "type": "photo",
                    "media_url_https": item.url,
                })
            })
            .collect())
    }

View on GitHub (pinned to 8600b91f42)