{"record":{"id":"aa499c385e753b27","repo":"tonhowtf/omniget","slug":"html-request-returned-http-mod","errorCode":null,"errorMessage":"HTML request returned HTTP {}","messagePattern":"HTML request returned HTTP (.+?)","errorType":"http","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"src-tauri/src/platforms/twitter/mod.rs","lineNumber":1048,"sourceCode":"        Ok(Self::media_info_from_twitter_media(\n            filename_base,\n            twitter_media,\n        ))\n    }\n\n    async fn request_html_media(&self, url: &str) -> anyhow::Result<Vec<serde_json::Value>> {\n        let mut request = self\n            .client\n            .get(url)\n            .header(\"User-Agent\", USER_AGENT)\n            .header(\"Accept-Language\", \"en\")\n            .header(\"Referer\", \"https://x.com/\");\n        if let Some(cookie) = Self::auth_cookie_string() {\n            request = request.header(\"Cookie\", cookie);\n        }\n        let response = request.send().await?;\n        if !response.status().is_success() {\n            return Err(anyhow!(\"HTML request returned HTTP {}\", response.status()));\n        }\n        let html = response.text().await?;\n        let items = Self::extract_html_photo_items(&html);\n        if items.is_empty() {\n            return Err(anyhow!(\"No photo URLs found in HTML\"));\n        }\n        tracing::debug!(\"[twitter] html extracted {} photo entries\", items.len());\n        Ok(items\n            .into_iter()\n            .map(|item| {\n                serde_json::json!({\n                    \"type\": \"photo\",\n                    \"media_url_https\": item.url,\n                })\n            })\n            .collect())\n    }\n","sourceCodeStart":1030,"sourceCodeEnd":1066,"githubUrl":"https://github.com/tonhowtf/omniget/blob/8600b91f4246848bac346874daa9e61c1fc5677a/src-tauri/src/platforms/twitter/mod.rs#L1030-L1066","documentation":"Thrown by TwitterDownloader::request_html_media when the HTTP response from the tweet HTML page has a non-success (non-2xx) status code. The HTML fallback strategy fetches the tweet page and scrapes pbs.twimg.com photo URLs out of it; this error means the page itself refused the request.","triggerScenarios":"Calling get_media_info on a tweet whose GraphQL and syndication strategies already failed, causing request_html_media to GET the tweet URL with a Referer header (and optional auth cookie), and the server responding e.g. 404 (deleted tweet), 403 (blocked/age-gated), or 5xx.","commonSituations":"Deleted, suspended, or protected accounts; X requiring login for that tweet (no/expired auth cookie); rate limiting (429); corporate proxy or CDN blocking the request.","solutions":["Inspect the logged HTTP status: 404 means the tweet is gone, 403 means auth/blocking, 429 means rate limited.","Configure a valid, unexpired auth cookie so the request is authenticated.","Verify the tweet URL is public and contains media.","Add retry with backoff for 429/5xx statuses.","Fall back to an alternative extractor (yt-dlp) when the HTML strategy fails."],"exampleFix":"// before\nlet response = request.send().await?;\nif !response.status().is_success() {\n    return Err(anyhow!(\"HTML request returned HTTP {}\", response.status()));\n}\n// after: retry transient statuses before failing\nlet response = request.send().await?;\nlet status = response.status();\nif status.as_u16() == 429 || status.is_server_error() {\n    tokio::time::sleep(std::time::Duration::from_secs(2)).await;\n    // re-issue request or delegate to another strategy\n}\nif !status.is_success() {\n    return Err(anyhow!(\"HTML request returned HTTP {}\", status));\n}","handlingStrategy":"retry","validationCode":"// pre-flight: ensure the tweet page is reachable and we have auth\nlet status = reqwest::Client::new()\n    .head(url).header(\"Referer\", \"https://x.com/\").send().await?\n    .status();\nif !status.is_success() { eprintln!(\"tweet page pre-check failed: HTTP {}\", status); }","typeGuard":"fn is_retryable_status(status: u16) -> bool {\n    matches!(status, 408 | 429) || (500..=599).contains(&status)\n}","tryCatchPattern":"match request_html_media(&url).await {\n    Err(e) if e.to_string().contains(\"HTTP 429\") => {\n        tokio::time::sleep(Duration::from_secs(30)).await;\n        retry_limited(3, || request_html_media(&url)).await\n    }\n    Err(e) => Err(e),\n    Ok(items) => Ok(items),\n}","preventionTips":["Send a valid auth cookie with HTML requests to avoid 403 login walls.","Back off and retry on 429/5xx statuses before giving up.","Verify the tweet still exists (HTTP 404 means deleted) before scraping.","Respect rate limits; serialize scraping requests instead of parallelizing.","Update extraction code when X changes its page structure or endpoints."],"tags":["network","http","scraping","twitter"],"backgroundTag":"http-error-response","analyzedSha":"8600b91f4246848bac346874daa9e61c1fc5677a","analyzedAt":"2026-09-12T14:29:19.317Z","contentChangedAt":"2026-09-12T14:29:19.317Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}