{"record":{"id":"6be7a811734432ec","repo":"tonhowtf/omniget","slug":"html-request-returned-http","errorCode":null,"errorMessage":"HTML request returned HTTP {}","messagePattern":"HTML request returned HTTP (.+?)","errorType":"http","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src-tauri/omniget-core/src/platforms/twitter.rs","lineNumber":981,"sourceCode":"        Ok(Self::media_info_from_twitter_media(\n            filename_base,\n            twitter_media,\n        ))\n    }\n\n    async fn request_html_media(&self, url: &str) -> anyhow::Result<Vec<serde_json::Value>> {\n        let mut request = self\n            .client\n            .get(url)\n            .header(\"User-Agent\", USER_AGENT)\n            .header(\"Accept-Language\", \"en\")\n            .header(\"Referer\", \"https://x.com/\");\n        if let Some(cookie) = Self::auth_cookie_string() {\n            request = request.header(\"Cookie\", cookie);\n        }\n        let response = request.send().await?;\n        if !response.status().is_success() {\n            return Err(anyhow!(\"HTML request returned HTTP {}\", response.status()));\n        }\n        let html = response.text().await?;\n        let items = Self::extract_html_photo_items(&html);\n        if items.is_empty() {\n            return Err(anyhow!(\"No photo URLs found in HTML\"));\n        }\n        tracing::debug!(\"[twitter] html extracted {} photo entries\", items.len());\n        Ok(items\n            .into_iter()\n            .map(|item| {\n                serde_json::json!({\n                    \"type\": \"photo\",\n                    \"media_url_https\": item.url,\n                })\n            })\n            .collect())\n    }\n","sourceCodeStart":963,"sourceCodeEnd":999,"githubUrl":"https://github.com/tonhowtf/omniget/blob/8600b91f4246848bac346874daa9e61c1fc5677a/src-tauri/omniget-core/src/platforms/twitter.rs#L963-L999","documentation":"In `request_html_media` (src-tauri/omniget-core/src/platforms/twitter.rs:981), the Twitter platform fetches a tweet's HTML page as a fallback when the API path is unavailable. After sending the request (with Referer and optional Cookie headers), any non-success HTTP status causes this error, carrying the status code in the message. It exists so the caller (`native_get_media_info`) gets a clear signal that the HTML fallback scrape could not even retrieve the page, distinct from a parse failure.","triggerScenarios":"Any call to native_get_media_info for a Twitter/X photo URL where the underlying reqwest request to x.com returns 4xx or 5xx: tweet deleted or made private, account suspended, auth cookie expired/invalid, rate limiting (HTTP 429), or Cloudflare/proxy blocking the request.","commonSituations":"Expired or malformed cookies in the stored auth (Self::auth_cookie_string()), scraping a protected/deleted tweet, X.com serving 403/429 to non-browser traffic, corporate proxy or geo-blocking, or X changing endpoints so old URLs 404.","solutions":["Log the HTTP status from the error message and handle it: refresh or correct the auth cookie if 401/403, back off and retry if 429, verify the tweet URL exists if 404.","Re-authenticate: update the cookie used by Self::auth_cookie_string() with a fresh logged-in session's cookie string.","Verify the tweet URL is public and still live by opening it in a browser (or curl with the same headers) before retrying.","Add retry-with-backoff around request_html_media for transient 5xx/429 statuses.","Check network egress (proxy/VPN/firewall) if the same request succeeds from a browser."],"exampleFix":"// before: single attempt, no status-specific handling\nlet response = request.send().await?;\nif !response.status().is_success() {\n    return Err(anyhow!(\"HTML request returned HTTP {}\", response.status()));\n}\n\n// after: refresh cookie and retry once on auth failure\nlet response = request.send().await?;\nif response.status() == reqwest::StatusCode::UNAUTHORIZED\n    || response.status() == reqwest::StatusCode::FORBIDDEN\n{\n    Self::refresh_auth_cookie().await; // re-login / rotate cookie\n    let response = request.header(\"Cookie\", Self::auth_cookie_string().unwrap_or_default())\n        .send().await?;\n}\nif !response.status().is_success() {\n    return Err(anyhow!(\"HTML request returned HTTP {}\", response.status()));\n}","handlingStrategy":"try-catch","validationCode":"// Rust: probe the tweet URL before invoking the library\nlet client = reqwest::Client::new();\nlet probe = client.head(url)\n    .header(\"Referer\", \"https://x.com/\")\n    .send().await?;\nif !probe.status().is_success() {\n    anyhow::bail!(\"precheck failed: tweet page returned {}\", probe.status());\n}","typeGuard":"fn is_http_ok(status: u16) -> bool { (200..300).contains(&status) }","tryCatchPattern":"match native_get_media_info(url).await {\n    Ok(info) => use_media(info),\n    Err(e) if e.to_string().contains(\"HTML request returned HTTP 429\") => {\n        tokio::time::sleep(Duration::from_secs(30)).await;\n        retry_with_backoff(url, 3).await\n    }\n    Err(e) if e.to_string().contains(\"401\") || e.to_string().contains(\"403\") => {\n        refresh_twitter_cookie(); // re-auth then retry once\n        retry_once(url).await\n    }\n    Err(e) => { log::error!(\"twitter fetch failed: {e}\"); show_user_error(e); }\n}","preventionTips":["Keep the stored Twitter/X auth cookie fresh; rotate it before expiry.","Only pass URLs to public, existing tweets — check the URL in a browser first.","Respect rate limits: throttle scraping requests and back off on 429.","Set browser-like User-Agent/Referer headers to avoid bot-blocking responses.","Monitor status codes from the error message and branch handling per class (auth vs rate-limit vs not-found)."],"tags":["network","http","twitter","scraping"],"backgroundTag":"http-error-response","analyzedSha":"8600b91f4246848bac346874daa9e61c1fc5677a","analyzedAt":"2026-09-12T14:29:19.317Z","contentChangedAt":"2026-09-12T14:29:19.317Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}