{"record":{"id":"7672731d4308df4c","repo":"tonhowtf/omniget","slug":"no-photo-urls-found-in-html","errorCode":null,"errorMessage":"No photo URLs found in HTML","messagePattern":"No photo URLs found in HTML","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src-tauri/omniget-core/src/platforms/twitter.rs","lineNumber":986,"sourceCode":"\n    async fn request_html_media(&self, url: &str) -> anyhow::Result<Vec<serde_json::Value>> {\n        let mut request = self\n            .client\n            .get(url)\n            .header(\"User-Agent\", USER_AGENT)\n            .header(\"Accept-Language\", \"en\")\n            .header(\"Referer\", \"https://x.com/\");\n        if let Some(cookie) = Self::auth_cookie_string() {\n            request = request.header(\"Cookie\", cookie);\n        }\n        let response = request.send().await?;\n        if !response.status().is_success() {\n            return Err(anyhow!(\"HTML request returned HTTP {}\", response.status()));\n        }\n        let html = response.text().await?;\n        let items = Self::extract_html_photo_items(&html);\n        if items.is_empty() {\n            return Err(anyhow!(\"No photo URLs found in HTML\"));\n        }\n        tracing::debug!(\"[twitter] html extracted {} photo entries\", items.len());\n        Ok(items\n            .into_iter()\n            .map(|item| {\n                serde_json::json!({\n                    \"type\": \"photo\",\n                    \"media_url_https\": item.url,\n                })\n            })\n            .collect())\n    }\n\n    async fn try_graphql(&self, tweet_id: &str) -> anyhow::Result<Vec<serde_json::Value>> {\n        let token = self.get_guest_token(false).await?;\n\n        match self.request_tweet(tweet_id, &token).await {\n            Ok(json) => Self::extract_graphql_media(&json, tweet_id),","sourceCodeStart":968,"sourceCodeEnd":1004,"githubUrl":"https://github.com/tonhowtf/omniget/blob/8600b91f4246848bac346874daa9e61c1fc5677a/src-tauri/omniget-core/src/platforms/twitter.rs#L968-L1004","documentation":"In `request_html_media` (src-tauri/omniget-core/src/platforms/twitter.rs:986), after a successful HTTP fetch, the HTML is parsed with `extract_html_photo_items`; if it yields zero photo entries this error is thrown. It means the page was retrieved but the expected photo URL patterns could not be extracted — X.com's markup changed, the page is a login/consent wall, or the tweet genuinely has no photos.","triggerScenarios":"Calling native_get_media_info for a Twitter photo URL where the fetched HTML contains no extractable photo items: tweet is a video-only or text-only post, X returns a login wall/'Something went wrong' page, or DOM structure changed so extract_html_photo_items' selectors/regexes no longer match.","commonSituations":"X.com frontend redesign breaking the extractor's patterns, logged-out scraping hitting the guest/login interstitial, pointing the tool at a non-photo tweet URL, or Twitter serving a JS-shell page without embedded JSON photo data to the request's User-Agent.","solutions":["Confirm the URL actually points to a tweet containing photo media; text/video-only tweets will never yield photo URLs.","Log/dump the fetched HTML (tracing::debug) and update extract_html_photo_items' patterns to match the current X.com markup.","Send the auth cookie so X serves the full logged-in page instead of a login wall — check Self::auth_cookie_string() returns a valid session.","Set a realistic browser User-Agent alongside the Referer header so X doesn't serve a stripped shell page.","Fall back to the primary API/syndication path instead of HTML scraping if the extractor is stale."],"exampleFix":"// before: silent empty result\nlet html = response.text().await?;\nlet items = Self::extract_html_photo_items(&html);\nif items.is_empty() {\n    return Err(anyhow!(\"No photo URLs found in HTML\"));\n}\n\n// after: diagnose wall vs. real no-media\nlet html = response.text().await?;\nif html.contains(\"log-in\") || html.contains(\"Enter your password\") {\n    return Err(anyhow!(\"Twitter returned a login wall; auth cookie missing or expired\"));\n}\nlet items = Self::extract_html_photo_items(&html);\nif items.is_empty() {\n    tracing::warn!(\"[twitter] no photo entries; html_len={} (extractor may be stale)\", html.len());\n    return Err(anyhow!(\"No photo URLs found in HTML\"));\n}","handlingStrategy":"validation","validationCode":"// Prefetch and sanity-check that the tweet actually contains photo media\n// (e.g. via the syndication API) before calling the HTML-scrape path:\nlet meta = client.get(format!(\"https://cdn.syndication.twimg.com/tweet-result?id={id}\"))\n    .send().await?.json::<serde_json::Value>().await?;\nlet has_photos = meta[\"mediaDetails\"].as_array()\n    .map(|a| a.iter().any(|m| m[\"type\"] == \"photo\"))\n    .unwrap_or(false);\nif !has_photos {\n    anyhow::bail!(\"tweet {} has no photo media; skip HTML scrape\", id);\n}","typeGuard":"fn has_photo_entries(html: &str) -> bool {\n    html.contains(\"pbs.twimg.com/media\") // quick marker that photo URLs exist in the page\n}","tryCatchPattern":"match native_get_media_info(url).await {\n    Ok(info) => use_media(info),\n    Err(e) if e.to_string().contains(\"No photo URLs found in HTML\") => {\n        // HTML extractor is stale or page was a login wall: fall back to API path\n        match native_get_media_info_via_api(url).await {\n            Ok(info) => use_media(info),\n            Err(inner) => report(\"neither HTML scrape nor API returned media\", inner),\n        }\n    }\n    Err(e) => report(\"twitter fetch failed\", e),\n}","preventionTips":["Send a valid auth cookie and browser-like User-Agent so X serves the full page, not a login wall.","Verify the tweet contains photo media (not video/text-only) before scraping.","Log the fetched HTML length/content on failure to detect markup changes early.","Pin and regularly update the extractor patterns against X.com's current DOM.","Prefer the syndication/API path when available; treat HTML scraping as a last resort."],"tags":["parsing","scraping","twitter","empty-result"],"backgroundTag":"empty-result-set","analyzedSha":"8600b91f4246848bac346874daa9e61c1fc5677a","analyzedAt":"2026-09-12T14:29:19.317Z","contentChangedAt":"2026-09-12T14:29:19.317Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}